Carto felt the same. This is an achievement for a 2d puzzler, putting it alongside Journey or Thomas Was Alone in its focus on relationships at the center of the story. The game tells the story of a young girl named Carto who gets separated from her grandmother on a grand adventure when a storm collides with their airship. Carto has to travel through many foreign lands and cultures to find her way back to her grandmother.

The Mechanics

Carto, the game, accomplishes this through a tile laying mechanic similar to the one found in the board game Carcassonne. The only rule is that tiles have to be matched against tiles with similar edges: water to water, grass to grass, road to road, etc. As you walk around the different domains you’ll discover more tiles to lay down that will move the story forward. Most of the puzzles in the game require manipulating the tiles in unexpected ways to unlock new areas, or combining them in a specific pattern. This will unlock new tiles or reveal new characters. It’s hard to go into too much detail here without spoiling them, but I found them enjoyable and novel to discover.

I only found myself on Google once to get an answer I needed. I appreciated that the game was not here to make me overcome a daunting task, or unlock a specially hidden combination in order to continue the story. Solving the small puzzles made me feel smart in a warm-fuzzy sort of way instead of a conqueror of the world sort of way. And that felt right for the aesthetic of the game.

Carto managed to fit a lot of story into the actions I took as a player. I remember trekking across the desert, sliding around the ice caps, and jumping up down and around a volcano because the ways I manipulated the tiles matched the aesthetic of the location. It reminded me of Breath of the Wild, where the vast open spaces aligned with the melodies and the mechanics to reinforce the wistfulness and the feelings of emptiness. For Carto, the tiles, fun writing, and mechanic reinforced playfulness, adventure, and community.

The Story of the Game

What’s great about the story of Carto is that it doesn’t have the main character face an evil character who’s trying to take her new friends or her grandmother away. There’s no apocalypse, conquering, or drama to be found, just a problem to solve. She meets a challenge that is familiar and mundane (a storm) and she has to work with other people to overcome it. That doesn’t make the puzzles any easier, but it centers the story with more grounding. The different people she meets help her on her quest, exploring the world, but she (and you as the player), must solve the puzzles on her/your own to progress.

There's nowhere more clear with this than the first world, where you learn your soon to be friend is about to be sent away due to a tradition to explore and find a new home. She will never see her family again and is sad about this. A lot of the dialogue with her is about coming to terms with this fact, but without the angst of a typical “rebelling against and modifying tradition” or the shattered family trope with fighting or yelling. There is frustration, but everyone is open about their sadness and hope. As a result it's calm and uplifting even with a melancholy undercurrent.

There’s a mirroring here of the real world pandemic I suppose. The virus isn’t something to conquer or defeat. Even if we end up eradicating it, it won’t be through battle or capture. It will be because countless people cooperated and worked together to solve problems to produce vaccines that eliminate it. Obviously Carto’s challenge is much less severe, but the severity of staying home, of not seeing friends isn’t in a contest to win, it’s in mental perseverance and discipline instead of outward conquest. So Carto’s focus on an external challenge with communal support (support that in the real world has been stripped by the pandemic) felt similar in the problem but comforting in the warmth of interpersonal experiences that aren’t accessible at the moment.

Wrapping it Up

This is a small spoiler, but there's a moment at the end that speaks to the cleverness of the design. As you get ready to wrap up your journey, the game asks you to revisit each space and “finish” the map, connecting all of the tiles in that space into a single unit. This isn't a hard problem as there are limitless combinations. But this small ask reminds you of the places you've been, and literally closes the loop on what the puzzles pieces you've been using into a single “place”. It was a short walk down memory lane that made me smile.

In a year that was oppressive, disconnected, and sometimes prone to despair, Carto was thoughtful, humane, connecting disparate elements of what I love about storytelling, human connection, and puzzle design. It's available for PC, PS4, and Nintendo Switch. It's one of the few games I don't think there's a type for. You should go buy it and have fun on an adventure.

Dicey Dungeons has a deceptively simple design. Each turn you roll dice and use them to activate equipment, damaging your enemies. Some equipment deal damage based on the number of pips on the dice, others require even/odd numbers on the dice, and apply status effects. There is a clear cause and effect between the equipment you use and the damage you deal to each enemy which made Dicey Dungeons easy for me to pick up. This initially led me to assume that the design under the hood was equally straightforward, but as the designer Terry Cavanaugh detailed in a blog post, there’s a lot of complex work going on just beneath the surface. This is a consistent theme in the game, rule complexity is hidden so that the player can feel the flow.

The challenge of the game comes from picking the combination of equipment to battle with. Players will find some equipment in chests which they can get to after defeating monsters, some will be found in “carts” either in exchange for money or for other equipment they own. Getting these decisions right feels like the key to winning the game: pick the wrong equipment and even the best rolls won’t save you.

The Delight of The Dice

Where other games hide the math behind damage calculation, Dicey builds the entire game around showing players how damage is calculated. Character health levels are counted in the 10s (except for the final boss), and most attack values are between 1 and 6, the values found on the dice you roll. The transparency makes the game feel more approachable than other rogue likes. You don’t have to spend a lot of time figuring out what the underlying system is doing, and the system is simple enough that you can reason about it in your head.

For many rogue likes the procedure of the world is designed and then players combine objects to exploit and create new possibilities no one imagined. Dicey Dungeons feels more deterministic and more authored than other rogue-like games. While the pathways are semi-random and the equipment you will find is randomized, there are a set number of levels you can reach, and certain classes of characters that you will meet on each floor. The “combo potential” of equipment interaction is limited by the number of dice you roll per turn. While this might limit the depth of the game, it also means there’s fewer unknowns to catch you by surprise, making the game overall feel more approachable. Instead, fun is unlocking new content and seeing what tweaks and twists the designer has laid out for the player. The tweaks and variations behind each door are the heart of the game’s depth.

Gentle Exploration

Part of the fun of playing a rogue-like is in building an understanding of how the world works. For example, in Spelunky HD you could try to steal something from the shop-keeper only to learn that the shopkeeper will try to run you down and kill you if you do. This form of exploration is fun, it encourages players to try stuff out, but also can hide a lot of the depth of games from players who don’t think of experiments to try. I’ve found confronting many different axes of uncertainty to be a barrier to my enjoyment of these games. By narrowing the types of exploration, Dicey Dungeons left me more free to fully explore the combinations the equipment and player powers have to offer.

Dicey Dungeons simplifies exploration by focusing on a discrete set of variables: how the different equipment interact with the dice roles and player powers to create strategy. The run lengths are short, so it’s easy to try out a set of equipment and then learn why they don’t work, only to immediately restart. And the reward for leveling is an increased number of dice, which means that as a run continues it’s more likely that your equipment will do what you want. This creates a tight feedback loop where a player tries to see how different pieces of equipment work together, loses quickly for negative feedback, or is rewards by getting to do more of the same thing they already enjoyed.

The reward structure also sets up a clever inversion during the harder levels (doors 4-6). Rather than only increasing your rewards, the game will start making it harder on you as you go further, potentially by adding tricky rules or by reducing your health as you gain levels. By this point though, I felt comfortable enough with the character, having a deep understanding of the equipment I preferred, to overcome the challenges the game threw at me, which increased my feelings of success. This is where the authored approach truly shined, it felt like the game had built me up by giving me support as I learned the character, and then turned the game upside down on me right when I was ready for a deeper challenge. I felt successful the whole way through.

The Long and the Short of It

The downside to this authored approach is that when I get stuck on a level, the challenges can feel arbitrary and painful. I found myself wondering if forcing the player to go through so many different concepts (battling with each character at least once, and winning with one character at least 6 times) create a barrier to the types of exploration that Dicey Dungeons tries to encourage? Completing a single round of Dicey Dungeons_is simple (about 30-40) minutes, but completing the entire game is an investment. This is the biggest barrier to the game, and it’s possible to imagine that it could dissuade some people from playing the game altogether. Terry Cavanaugh, the designer, even contemplated as much on twitter. He said he felt like _Dicey Dungeons is too long. In short, there’s a lot of game to get through, to get to the content. For a game that’s otherwise so approachable, the time investment to get satisfying results could be a turnoff.

I prefer Dicey Dungeons to most other roguelikes because the time to complete a single “run” (the level or campaign in the game) is 30 minutes, instead of the more typical hour or so. Compare this to Slay the Spire, one of the gold standards of the genre, where a successful run can take an hour or more. But in order to understand where Dicey Dungeons trips it’s important to talk about the motivations that games set-up their players with in playing. Film Crit Hulk, a popular film (and sometimes game) critic, has a good heuristic for different types of games. He groups them into roughly one of two categories: “flow” games, like Assassin’s Creed and “learning” games, like Slay the Spire. In flow games, the goal is to successfully engage the player while delivering content and mastery at well timed intervals to make the player feel and improve in line the content they’re delivering (often in this case story beats). Learning games are more about the system itself, playing is a means to deeper understanding of the system.

Dicey Dungeons lives somewhere in the middle of the two. Dicey Dungeons is a learning game in the sense that you are trying to learn how to work well with each character in order to complete a run. It is likely that you will fail as you learn which equipment work best, and which upgrades to use. As you fail you will get better at making decisions for each character. But it’s also a flow game in the sense that each new character is an exciting piece of content, and each piece of content is not so complex to operate that there’s a lot of excitement to fully exploring the possibilities of operating them. I found most of the excitement was in being given a new concept or idea to play with, rather than in a deeper understanding I got from thinking about the rules of the game in a new way.

Flow games can trip over themselves when story becomes locked behind challenges that require repetition to overcome. I believe the main reason for this is the overjustification effect. Overjustification is the concept that external rewards reduce intrinsic motivation to complete tasks. In this example, narrative (the game show) or content (new characters or challenges) are the external rewards and the joy of playing a mechanic is the internal motivation. Flow games often rely on the designers to fine tune those moments where players might get stuck and offer them a helping hand. This might come in the form of (sticking 5 shovels around an island to make sure you find 1 in the case of A Short Hike), or taking control of the camera to point the player directly at the thing they need to see. But a learning game like Dice Dungeons doesn’t have this same control because the designer can’t “know” at any moment where the player might be.

When Dicey Dungeons is successful the hybrid of the two feels exhilarating. Winning a run after 3-5 attempts has all the exhilaration of a learning game, because I figured out the right combination of equipment and strategies to win, and also because I know I’m getting a new character, new challenge, new equipment to experiment with. The combination feels like a burst of energy. But it can start to feel like a slog when I’m torn between wanting to know what happens next but also need to focus on learning about the game so I can beat it. And because the mechanics are more authored, I felt less like I’m missing some underlying mathematical concept, and more like the arbitrary way a particular level was designed hurts. I found myself wishing the path to unlocking the story were easier, so I could spend more time picking the game up to explore new concepts rather than being driven by a desire to unlock the final character or the end of the game. It’s hard to focus on the joy of a single run when there’s a payoff waiting (the final boss) that you know you have to get to.

Games like Super Mario Odyssey have managed this by reducing the requirement to unlock new levels or new story to a small percent of the total content of the game and then designing the areas so that the players want to come back for more. I think Dicey Dungeons is a strong enough game to have done the same, and I found myself wondering why I needed to beat every character 6 times to unlock the final boss. I would’ve happily continued playing through unlocked levels even after, because the novelty of each new experience was so inviting. But after grinding through the game in 2019, I was so exhausted that it wasn’t until it came out for the Switch that I picked it back up. There’s nothing wrong with a player deciding to put down a game before it’s done, and even if you played each character once, I would still recommend this game. But games should also afford players to end at their pace without making them feel like they’re breaking the logic or the narrative of the game.

Unequivocally: You Should Try It

I still happily completed every level in Dicey Dungeons, and I picked the game up again when it came out on Switch because it brought me joy. Given that I tend to drop most games before experiencing more than half the content (the aforementioned Slay the Spire and Super Mario: Odyssey come to mind here), completing all of the challenges and defeating the boss represents an achievement for me. And it wasn’t because I felt like I had to, to experience the story: the game brought me novelty, excitement, and learning each time I played. There’s something inspiring to play through the results of a game designer experimenting with multiple different ways to twist and tweak a simple concepts: dice rolling. The love put into the design is easy to see as you play the game. The game also has a vibrant modification culture which the designer supports and encourages. Due to the restrictions of consoles, modifications are only accessible on PC, but if you have the PC variant I’d encourage you try to out the MegaQuest if you’re looking for more to play. Dicey Dungeons continues to feel like an exciting and welcoming possibility space that I expect to play off and on for quite some time.

I find myself asking this question because my grandmother passed away in January, ending the physical bond I shared with two of the closest people I had in my childhood. And as I reflected on what she meant in my life, the weekends I spent over with her and my grandfather playing checkers and eating grilled cheese. My grandparents were conservative down to their bones, both in their politics and their lifestyles (they re-used Ziploc bags). I was a touchy-feely kid who was dominated by my emotions (and pessimism). Games gave us a way to communicate and interact joyfully.

It feels ironic that while my relationship with my grandparents was defined by games I never found a way to describe my love of games in a way they could understand. I couldn’t communicate the feeling of being swept away on a gust of wind by the possibility and anticipation of a new idea or concept. The way I feel like I can feel the texture of a mechanic of a game or the fun that I have hearing an idea rattling around my brain. These delicate impractical things were hard to describe.

I still remember college when I tried grappling with the now tired concept of whether games are art or craft. It’s the type of discussion that feels fundamental to someone who loves games but alien to anyone from the outside looking in. In the face of reflecting on a person’s life that conflict can feel rather shallow. But I’m reminded about the book Cork Dork and how building expertise changes the way we experience our lives. And my experiences with games have deepened my appreciation of those experiences with my grandparents.

Arts and Crafts

The debate between art and craft, especially in American society, is about the difference between whether something is authored and worthy of intensive consideration, or whether it was designed to solve a problem and searching for meaning within it isn’t worth the effort. This is a gross simplification, but for a long time, game designers were looking for recognition as being “art” because it would mean they’d made it in being worthy of higher consideration. If you google “Games Citizen Kane” or search for Roger Ebert’s article on whether games are art, you’ll get a sense here.

This discussion might seem entirely disconnected from trying to make sense of a group of lives 2 decades ago, but it isn’t. If you believe that games should be analyzed only as art, you might come to the conclusion that the grilled cheese sandwiches that my grandmother made were disconnected from the experience of the game or that because checkers “wasn’t saying anything” that there’s not much substance there to analyze. And if you see a game as a designed element it cheapens the experience. It was a way to pass the time with my grandfather. But checkers was both and neither of those things.

What is Checkers

Checkers is a game played on an 8×8 grid, typically with red and black discs lined up in alternating squares on either end of the board. The goal of the game is to eliminate all the disks of opposing colors by jumping them. Each turn you either may make one more diagonally, or if there is a “jump” available, you must make it. If one of your pieces makes it to the other end (the opponent’s end) of the board it is “kinged” and may now move in either direction instead of only forward. This turns it into a powerhouse piece. The game continues until stalemate or one player eliminates all the pieces of the other player.

As a game, checkers feels pretty same-y each time you play. While there are plenty of different configurations to the board, the game itself doesn’t have much dynamism. Unlike in chess there aren’t interesting clusters or combinations of strategies that present themselves for a varied experience. I realized 2 things rather quickly: getting a King was great and the side spots just above your starting position were extra valuable because they could block off a lot of territory without risking their pieces and allow yours to advance. Checkers taught me that even obvious strategies can feel rewarding when they are expertly deployed.

A Problem to Solve

When we talk about games, we spend a lot of time talking about clever design tweaks that a game makes to make us feel smart. Or design tricks that challenge us and lead us. We don't spend all that much time talking about the way that we act on games and change them, and how the value of some games is how easily they can be changed. We call things "game systems" when they can be built on top of, like how a 52 card deck is a "game system". But the deeper you go in game design the more you recognize that play is this vulnerable special place where humans are free to experiment with new ways of expressing themselves and becoming vulnerable. Bernie DeKoven, one of the members of the New Games Movement, talked about this concept of the Play Community. In his reading, the games we play kind of form a contract and a space (some people call this the magic circle) that allow us to engage with other people in ways we might not otherwise.

Checkers was neither art nor design for me, it was both. Games are design because they solve a specific problem, which is how do you get two specific people at a specific time to play together. Thinking in terms of a community of play, a game is a contract that binds people within a space under terms they find agreeable. Games are art because the experience as the result of that play is beautiful and meaningful. It’s the type of stuff that sticks with you. Checkers was the contract my grandfather and I shared together, and it was guaranteed by the grilled cheese.

Checkers was, to put it tritely, “a game that could be played and appreciated by a 10-year-old and an 80-year-old”. Neither had a college degree, at least one had little patience and the other had a memory who failed him. The games fit. Finding that degree of fit between two or more people is hard. It’s an accomplishment that is worth celebrating because when that fit works it sparks pure artistic magic that is rare in my view. Moments when, regardless of your past or future paths, a brief understanding and appreciation can be passed between you.

Checkers was special because it was simple enough for a 10-year-old (honestly I can’t remember the exact age) to learn while he was getting destroyed by his 80-year-old grandfather. It was also straightforward enough for that grandfather to continue to remember even though his memory failed him consistently and at strange times that would confuse the young boy when he would have to remind his grandfather how the rules worked. I don’t know, at that time, how many other games would have fit that need. Something like Chess or even Cribbage would have likely been too complex for the young boy to learn or have too many rules for the grandfather to remember. The grilled cheese solved the problem that the boy was hungry and wouldn’t stop eating. The Italian grandmother also wouldn’t stop feeding him. The 2 grilled cheese days were extra special.

My first play community was my grandparents. While my grandmother never played the games, the grilled cheese she made and the conversation she offered as a spectator became an important part of the experience. Those games of checkers, I think were what bridged the gap between an entirely emotional child and a pair of grandparents who had practiced being stern and unflinching, for whom emotional expression was something to withhold.

It Was Playful

I know both of my grandparents were tough, demanding people, to put it gently. As a kid, you can occasionally catch the things that your parents let slip, that they still try to hide from you as you get older, to protect those perfect memories you formed. But it would be rather hard to corrupt those memories: I know many of my friends didn't have grandparents around at all, much-less ones that lived next door. But the games of checkers were gentle. The sternness might appear elsewhere, if I were being asked to complete a task. Playing checkers I might be gently chided, "Stuie you have to pay attention to the game", as I was distracted by the television sitting behind me.

And even though as a family we prided ourselves on doing things correctly, well, we didn’t play checkers with all the right rules a single time from what I can remember. We never used the rule where you had to jump an available piece. It didn’t make sense to us, so we didn’t use it. I think this is also some of the magic of games. We can reshape them to fulfill our needs without changing them conceptually. We were still playing checkers, just our own version on it.

The checkerboard told a story. It started life inside a red cardboard box. The box got opened and closed so much that it fell apart, so I started storing the pieces in a Ziploc baggy and carry over them and the board without protective casing. And then the board split because it had been opened and closed so many times, so we put striped packaging tape down the middle. And then a piece or two got lost, so they were replaced with my ever present legos. The way the pieces contain the memories of those experiences still inspires me.

Making Meaning

Checkers was also my first real brush with mortality. Unlike other areas, like my parents ever-present concern about his health or his strokes (which given his survival rate for the first 2 decades I knew him created a very incorrect vision of what having a stroke meant), seeing him fail to remember something in checkers, or start to lose more often felt tangible and real. I could see he couldn’t do things he used to be able to previously do. The game made it tangible.

It seems weird to celebrate games for their ability to recognize our mortality, but that’s part of what art does. It can make an abstract concept like death tangible. I suspect this is part of what made me so enthralled by games as an intellectual pursuit. Checkers didn’t set out to focus on mortality, but it ended up giving me the space to consider it and to appreciate the connection I had at the moment.

I think that's why I equate two player games to a higher level of appreciation than games that play with three or more people. I equate them directly with love. Because communication that didn't come easily to me or my grandfather the movements of the pieces substituted. The games on the weekend felt intimate without being forced.

Possibly Predictable Magic

This is why games make such excellent design objects. They solve problems (connecting people) and they are craft objects designed to be manipulated and transformed to suit the needs of their environment. Matching games to people in the right way helps give our communities of play a starting point to leap off from, to change and shift, and the starting point is what defines whether that’s possible and where it might go. Checkers wasn’t special because it was old or because it was a perfect design. It was special because both of us could learn, understand, and enjoy it on terms we were comfortable with. A lot of the way I view games is defined in part by the simple act of setting down a checkerboard in front of my grandfather every weekend.

But we changed the games to be more to our liking. The subtle shifting of the rules of games is so banal that we take it for granted. Those small changes, the ones that the system of the game left open for us made it possible to connect the mental gaps between a young boy and his grandfather. Neither my grandmother nor my grandfather would have used the word, but we were a community centered around play, around the playing of games and around the watching of games (football in particular, sometimes baseball). And we shifted the games we played to meet our needs.

As his dementia took hold, the last few games with my grandfather were of Gin Rummy. Again we played the game with “incorrect” rules. By this point I would occasionally hold back or lose concentration to let him win or be competitive. It wasn’t so much for him as it was for me to tell myself a fiction that his memory wasn’t getting worse or failing. But by the time I reached high school the grilled cheese and the games discontinued. The relationship we had built was strong enough that it could survive without them.

Checkers is not a uniquely great game. Neither is Gin Rummy. And grilled cheese has little to do with an interesting narrative or mathematical mechanic. But these three things are inextricably linked to my emotions and connections to games. You see, for close to a decade of my life I spent at least a day of almost every weekend playing checkers (and then Gin Rummy) with my grandfather who lived next door while my Grandmother made me a grilled cheese sandwich.

In Play Anything, Ian Bogost talks about how what games are special at doing is inverting our expectations by pulling the background forward and pushing the foreground (the things we take for granted) to the back. The benefit here is to make special those things we don’t appreciate or too often take for granted. But with board games, the stuff being foregrounded isn’t the games, it’s the people we spend time with (and ourselves). And what makes each game special is the way it draws out certain elements of our experiences and our relationships to other people in interesting ways. The important thing here isn’t that Checkers or Grilled Cheese or Gin Rummy are special. What was special was the way that Checkers and Grilled Cheese solved a problem of connecting generations and emotions into something that became meaningful, and maybe even profound for my life.

Finding a game that connects a group of people is magic. It’s as simple and mundane as that. If movies are about capturing and storing the magic of a moment, and books are about building beautiful mental palaces, and music is about breathing an idea through a rhythm, games are a box of tinder that can spark a new moment of magic. The value in reviewing them is about helping people understand when and for whom each game might solve a problem and making it easier for more people to discover that magic.

In the summer of 2020, I worked with the finance team at Pana to improve the reliability of our system, which matched booking charges to trips to bill them back to the customer's trips. The system generates a transaction for every request which meant extra transactions floating around unassociated to a trip, making it difficult to understand which were errors and which were "correctly" unassociated. The system occasionally duplicated transactions or failed to create a transaction for a trip. Because of the "floating" transactions, it was hard to tell if this was because it failed to generate or because it generated but didn't link. And, because of how the charging worked, occasionally we would bill customers before the transaction finalized, resulting in occasional "correction" charges of small amounts added on to the customer.

The result was a system that was unreliable and hard for our financial team to use (to validate we knew where our money was) and reduced trust from our customers who had to follow up with us to validate that we billed them the correct amount.

In addition, while we used three key services at the time (Uber for rides, Emburse for virtual cards, and Transferwise for reimbursements), there were plans to expand the system to further providers. We wanted to develop something that was extensible and customizable, reducing work for future developers to implement a new payment type.

The Plan

We centralized around a few different core requirements that the system should be able to handle and continue operating on.

Terminology

Working with the finance team, we came up with a set of vocabulary and system to

• ingestion: storing a flat copy of external financial data sent to us by a provider
• application: attempting to save the external data as a Transaction record
• charge: charging the customer for the applied transaction

We actually built out an overview of how the tool worked in Whimsical, which helped us iterate on our ideas about how the system functioned and how we would like it to function.

Multiple Entry Points

Each of the external systems had different ways of communicating with us. Emburse used a webhook that would send us updates anytime a transaction state change occurred, Uber sent us a monthly csv, and Transferwise sent the result in the response. The system we built had to accept entry points from a variety of sources, as we couldn't dictate the method used by our providers.

Duplicative Data

Because of the external sources and multiple entry points, duplicative data could come through. We wanted to capture each instance of that input but also only store the correct number in our financial system at the end. This meant having a unique primary key per external source that the system could use to track and keep a record of each instance. This reliability also had the benefit of meaning our internal team could "rerun" a CSV upload multiple times to validate it went through, and it also gave our reports a historical record of each input attempt for free.

Outcome Record

The system had to produce a record of what happened when it tried to apply a transaction. This had the benefit of making it easier for on-call developers to debug issues if they came up (which helped us as we were running initial tests on the system) and also support reporting tools for the financial teams.

Reporting Data

The financial team wanted to build queries and reports to determine the percentage of successful attempts and review any areas of concern. They also wanted a set of tools to determine the total amount of outstanding spend for a period of time.

Stingy Restrictions and Replayability

From a technical perspective, the other goal we had was to restrict the amount of data entering the system to the minimum amount. This would help us debug the system more effectively, and because the system persisted the external data, we could theoretically push a code change, and "replay" the application of a detail that had an incorrect outcome.

The System

Processors and Backups

We divided the system to ingest and apply transactions into (roughly) three separate processors. The first processor takes a single external record and saves it for processing. The application processor accepts a source and a record id and attempts to generate a new transaction or invoice for the customer if necessary. There's also a third processor, built as a cron to check for any records in the table that have gotten lost (for whatever reason) and then attempt to apply them.

Custom entry points kick off the ingestion processor, which kicks off the application processor. The backup cron also kicks off the application processor if it finds any stragglers hanging around.

Rough Architecture

One goal from a technical perspective was to reduce the amount of custom logic that each external provider used. At the time we had 3, but it was possible that as we grew, we'd want to add multiple different types, with a different configuration for each.

Typescript came into handy here from a perspective of Class and Interface management. We created an Abstract class that contained the bulk of the functionality but then use types to require each of the customized methods to return the same details.

This meant each tool could compare things like status (which would differ between external sources) with the same application processor. It was the marriage of extensibility without repeating ourselves that we were looking for.

The application processor is where all the magic lies. It checked if it still needed to attempt processing, and then would try to find an appropriate linkage to a trip and customer to bill. If anything unexpected occurs along the way, it would store an error record so that someone from finance could follow up. In all cases, it would return attempt details, so that we could explain what had happened.

The final decision we made was to wrap all creation logic (any new records that might need to be generated for charging) inside a single database transaction so that if any component failed, it would generate none of them. We didn't want to deal with any lingering records we had to chase down.

The Outcome

The result has been satisfying. The system has a 99.99% automated matching success rate, there's reporting built into Periscope that our financial team can use, and we've only received one bug ticket in the last 6 months, caused by an unexpected charge being sent to us from an external source. Because of the architecture of the system, we resolved that issue in under a day. Decoupling the system also allowed us to add custom error monitors to each component, so we can more easily tune alerts to each part.

He notes that so much of our discussion of productivity is framed in personal terms instead of business or systemic terms. He argues that from a business cultural perspective, our spirit of reinvention and experimentation is lacking when it comes to our workflow.

There are a few concepts I found most helpful in the book. Newport's term the hyperactive hive mind talks about the idea that companies have traded individual effectiveness for a sort of hive mind of knowledge that can act effectively, but slowly. He argues that this is bad because of something he calls the attention capital principle. His argument is that attention capital is our ability to turn information into value via focus.

The Distinction Between Workflow and Work Execution

I think part 1 of A World Without Email is worth reading. It solidifies a lot of concepts around productivity and systemic thinking. Part 1 of the book deals with making these connections between how the human brain is wired to deal with socialization in specific ways, and how our modern information workflows become the worst of both worlds, feeling like we have to be always on and always responsive. In part 2, he starts talking about specific workflows, centered around software workflow concepts of scrum and agile, specific recommendations for how to structure work in uncertain environments.

He wants to push back on the popular thinking that Peter Drucker articulated: the idea that all knowledge work must be autonomous. The two component parts that Newport breaks knowledge work up into are workflow and work execution. He calls workflow making up all of the ways that we communicate and work together and work execution is the heads down time. I can understand the desire to create this level of abstraction. It protects the autonomy that knowledge workers care about while also exposing some piece of it to outside management. But I think it gets at the wrong problem.

This sounds nice but... there isn’t actually a clear line between heads down making and heads up reviewing work. It’s a best practice for every piece of code that gets committed to go through a “PR” (named for Pull Requests or Peer Reviews) process. This is an opportunity for other developers to take a look at the code and comment on any changes. The issue here is that we’ve quickly turned a “value add” function into a “workflow” function. The developer can’t continue with other work because they need a review, either they have to ping people to get answers or they have to start something else, potentially building new context before switching back. As this article on pull request explains the pull request workflow causes serious gaps in the process. It's exactly the type of thing that gums up our working time.

Take a similar issue with a junior engineer who might be new to the team. Often it’s important for them to work on their own so they build the types of research and problem solving skills that will serve them as they grow in their career. Often the advice is “if you get stuck for more than 30 minutes, seek help”. Cal would seem to frame this as an inconvenience issue. The junior engineer could wait for a senior to become available. But, having been on both sides of the pattern, often getting help in that moment is important. If the junior has to wait longer it can cause frustration to ruminate on why they’re unable to solve the problem, or spiraling through unhelpful paths that result in more work for the senior developer to help untangle.

One solution to both of these issues (and one that gets explored in chapter 7 of A World Without Email), is to decide to do “pair programming” or “mob programming”. This is a process where 2 or more developers (with various types of structure) will decide to work together to solve a problem. As Google’s standards say, if code is paired on, it doesn’t need to be reviewed because the review has been part of the process. This sounds great, but has turned a “workflow” problem in to a “work function” problem. In order to make the system better we've made a change to how people do their work.

Narrowing Too Quickly

This isn't to say that we shouldn't try to negotiate and create clear principles for how communication occurs, it's to complicate this notion that we can abstract out the various parts of knowledge work into value add or communication time. One of the reasons it's called attention capital isn't just because our attention adds value, it's also because the value of information is in its ability to capture other people's attention as well.

A World Without Email narrows in on a process solution that programming has had for a while: scrum and agile concepts. The idea here is that because engineer's time is so valuable we should put strict delimiters on when work starts and how it's communicated. Broadly speaking it's a good starting point for small teams to work effectively together, but it doesn't solve the information overload problem.

Early days (2018) at Pana we had a problem with bugs in the system. Like every other high growth startup we were trying to work quickly to put stuff into the world and sell to new customers. Often this would result in a flood of bugs coming in taking up precious developer time to track down and fix. We put in plenty of systems in place to try to limit these, but the underlying issue was the stability of the system, the level of customer dissatisfaction that other business stakeholders would accept, and the knowledge of the system each developer had. We used scrum, but scrappy customer success managers (rightly) pushed through the process to get the solutions they needed to do their jobs.

This isn't to say that you should ignore the scrum and agile concepts discussed in Part 2, just that I don't think they alone will solve overload issues. There's a larger, more radical implication in A World Without Email that I wished got discussed.

We Don't Plan Enough

The pronouncement here shouldn't be for knowledge workers to implement simple queueing systems to take control over their days, it's for leadership and management to take a more active role in understanding who, how and when people in their organization communicate. A World Without Email reframes communication as a competitive advantage. Getting informational flow right means speed in execution and reduced stress for the participants.

But the implications of how and why to get communication flows right gets muddled a bit. In part 2, Newport talks about how a group of small businesses used systems like these to improve the lives of their employees. These are good examples of businesses with a lot of alignment in terms of priorities and a relatively small number of competing internal stakeholders. Doing this stuff at larger organizations or organizations that want to move fast doesn't just require better systems, it requires rethinking project planning and interdepartmental relationships altogether.

Often when we conjure the issues with big companies we think of countless hours in meetings planning. But I would argue these meetings involve little actual planning. Instead of planning and researching we typically quickly move to decision and risk limiting. Given a direction, how can we make sure the product we want to build actually happens. But the solution to this is paradoxical, as the book mentions. In order to get that speed, we need to spend more time up front planning and coordinating goals for projects as well as the communication patterns we intend to use to execute them.

This is where Henry Ford’s "spirit of innovation" could be productively injected into the system. As explained by this now famous IDEO principle, the oft discussed nurse workflow redesign seems like a perfect example of the workflow and work function changes that A World Without Email espouses. At the time, we liked to call this "Design Thinking" but this type of collaboration between business stakeholders and work functions to create systems that improved the functioning of the business are exactly the type of nuanced plans that we need to get out of information hell.

Aligning Incentives

Spending extra time thinking about how we work feels like a drag on productivity or a symptom of process overload. But the goal here isn't to create processes for their own sake's, it's the concept that we need to apply this sort of problem solving to our businesses in addition to the services we provide.

In order to make these sorts of behaviors permanent, we need a better set of measures and incentives that frame this sort of work is important. My sense is that companies fear the cost of doing the wrong thing more than they fear the cost of lost productivity which the attention capital principle shows is a backwards way of thinking. A single process cannot make up for this incentive misalignment. And the rate of process consideration has to match the rate of change at the business. A single process change will crumple if it's met with changing conditions and minimal will to maintain them.

While A World Without Email is a starting point for the conversation, more work is necessary to find, examine, and incentivize measuring communication and productivity as the goal of a business rather than the goal of the individual.

Especially in engineering, when we're searching for jobs we tend to look for high minded matches in terms of philosophy, attitude, and perspective on the world. I think these tend to be interviewing mistakes because the things that make us happy are simpler things like "are we shipping code all the time", "do we have customers who use our software?", "are we able to get positive feedback from those customers", "is there a rock competition that a random engineering manager runs at standup". These things have a much bigger impact on our day to day lives than things like "making a dent in the universe" or "are there interesting problems to solve". I'm not saying these aren't important, but... by the time I get to the interview stage of the process, I should have a pretty clear idea if my work will have a huge impact and whether or not it's an interesting problem to solve. It's either a a high leverage position (high rank or small team), and by the time I'm talking to someone who's not a recruiter I should have a pretty clear idea of the product or tool I'm going to be a part of building.

The goal of each stage of the interview is to determine if those ideals that I think I match on actually line up in practice.

How to Structure a Good Question

I have a few goals when it comes to asking good interview questions (as the interviewee). I don't always get this right and often I'll default back to the easy stuff, but at a high level I try to hit these marks:

1. Specific Examples: I want to allow the company to give explicit answers to the questions with examples that I can actually determine if they're true or false.

   • Asking something like "How do you feel about work life balance" is unhelpful here because it makes it hard to determine whether or not they're telling the truth. "We value it" might be a true sentiment, but it doesn't give any context on what work life balance means to them or get at the things I might care most about.

2. Not Too Specific: I want the question to orthogonally get at other deeper truths without being explicit about the ask.

   • Going back to the previous situation about work life balance, asking them point blank, "do you value work life balance?" or "do people work nights and weekends?" is also unhelpful because there's a clear right answer. Also getting too specific can be a problem "Tell me about a time you had to work more than 40 hours a week." might leave the interviewer without a solid answer (they're probably not tracking their time), which will likely result in the same initial vagueness I was trying to avoid.

3. Getting Off Script: I want to get a perspective on the company, but not from the default answers they always give, but closer to an honest interpretation of how they feel about the company.

   • The idea here isn't that I'm going to trick them into saying something bad, it's that often people default to the normal good stuff that is pretty common at every company. Asking a question that they haven't heard gets them to pause and recalibrate answers in a way that's productive for me.

4. Opportunities for Follow-up: The other goal is to ask questions that let me ask a follow up if necessary. It's not always (and given the limited amount of time, often isn't), but I want to set myself up for the possibility.

A Potential Example

If we use "Work-Life Balance" as the theme, I might ask a question like: "Tell me about a time you had to work hard in order to hit a commitment. How did you coordinate with the team and what did you learn from the process?" I like this type of question because it's open to the idea that sometimes teams will have to stretch to do things, but also asks them to articulate what they did in that situation, but without asking them if they'll give me the day off when I work an 80 hour week. I'm not boxing them into a corner with the question, but can hopefully get a good sense of how open they're being.

The Caveat

It's important to note that a lot of these might get more or less specific depending on the person, their tenure, and the role at the company. Each of these is a starting off point, but the goal here is to create a set of questions that connect the high level to the daily habits I'll act on.

The Questions

Culture and Mentorship

The goal with these questions are to get a sense of how they think about communication and team building to build strong teams that are fun to be a part of.

1. How do you think about hiring junior engineers?

• This is an important one to me because it signals a lot about how they view growth and professional development. One of my earliest bosses as an engineer made the point that teams with too many seniors tend to perform poorly because there isn't enough of a learning as an organization. A willingness to hire junior engineers also points to a solid nod to diversity at the company as it creates more pathways for people who don't fit a traditional engineer mold to join the organization. It also says something if they only hire juniors, because it might mean they look at engineering as a cheap resource.

2. What steps have you taken to improve diversity and inclusion?

• I want to hear about tangible things they've done to improve diversity and inclusion, especially how they measure it. I don't like asking "how do you measure" because it instrumentalizes the question and puts leaders on the defensive. That said, there are definitely good answers I'm looking for here: actions, partnerships, and metrics they're using to understand success. Bad answers tend to look more like admitting that they have a problem, or expressing a desire for "viewpoint diversity". (It's not to say that viewpoint diversity is bad, but that it's a completely different topic of conversation)

3. What differentiates a good engineer from a great one?

• This one I got from a close friend (who's written for this website). On its surface this is a question about skill and technique, but the deeper question is basically "what do you value in the people you look up to". It gets at asking questions about values and aspirations.

Product

As an engineer I'll be working with product managers and other stakeholders to ship code. Often companies will prioritize engineering happiness because of our expense and our relative value in the industry. Product often sits on the other end of the spectrum, answering directly to stakeholders unless there's a champion on the executive team who believes in independent and autonomous product teams.

1. How do product and stakeholders work together to set priorities?

• The issue here (that I've seen in my career) is that if product doesn't have autonomy to be effective at their jobs, it often also limits what engineers can do, because they're partnering people without the authority to make decisions. I want to get a sense of how they talk about product work (in terms of features or problems) and who has the rights to prioritize them.

2. How many production bugs do you have and how do you manage them?

• I started asking this one recently, and the responses have been really interesting. In general it depends on the size of the organization in addition to their practices. It also gets at how open they are about discuss production issues and who is responsible for fixing them. The challenge here is that sometimes this gets interpreted as "Are you going to throw me into a stressful on-call cycle". I'm still working on how to frame this more effectively.

3. What type of work cycles do you work in?

• How often work gets pulled in and shipped shapes a lot of how projects get scoped and developed. I like to get a sense of the cadence that they work in and how often they're learning. A good follow up question here might be: "Tell me about a really good retro and what you took away from it" to dig into how they're reflecting and improving their process.

Dealing with Issues and Specifics

1. What gives you the most angst in your day to day on the job?

• This is one that will let them flinch a bit and answer in a way that's probably fairly personal rather than "there's this crappy process we keep using", but the way they flinch should be informative to what they're aiming for. One interviewer responded with "I worry about making the right decision/technical design." This is informative because it seems like there's probably a high standard for work at the organization.

2. How do you give feedback to one another?

• I know earlier I said not to be blunt, but I think this one is good to be up front about. It's important to know whether or not constant feedback is the name of the game, or if you should wait for your yearly/quarterly review to get information about your performance.

3. I would like to grow to title [insert title here] how would I do that?

• This one really depends on the size of the company and your personal goals. If you're not looking for a specific career growth, there's no need to ask, but if you are, it's worth understanding what time scale they might offer that to you.

4. What types of metrics do executives share with the company?

• I'd be up front about this one. At most of the companies I've worked at, they've been very transparent with specific data about the targets we're aiming for, where we're at and how we intend to get there. If a company isn't doing that for you (especially as a software engineer), it's probably worth thinking about how set up for success you're going to be.

5. Talk to me about a mistake you made recently

• This one is blunt on purpose. It's similar to the production bugs in the sense of it's kind of a "show me your warts" question, but it's also trying to get them to be vulnerable about how they frame personal success and failure. Do they admit any fault? Do they admit to an inefficiency that's not really a mistake but a matter of happenstance? It's not necessarily bad if they do one or the other, but it's an extra data point in how they think about the world.

6. Tell me about a time you had to work hard in order to hit a commitment. How did you coordinate with the team and what did you learn from the process?

• This is the one from above. It gets at both personal and company perspective and also gives me the opportunity to ask follow-ups like "How often does this happen" or "How do you feel about it?" if necessary.

Pay Attention to the Types of Questions They Ask

In addition to asking the questions, it's important to get a picture of what they care about and what they're aiming for. Are they following up to anecdotes I've mentioned or are they walking through a list? Are they looking for a set of traits that they're checking off, or are they curious and trying to get to know me better as a person.

There's No Silver Bullet

These questions are meant to be a starting place to think about questions. I think all interview questions should be updated over time as attitudes about the hiring process involve. Certain questions will fall in and out of favor (which means people might be more or less practiced with an answer) and the important thing isn't to instrumentalize the process or assume there's a single set of questions to ask to tell 100% of the time.

I hope this helps!

So now that we've got here, what do we do?

Well, if you're someone who's really interested (and has the time and money) in becoming a top 1% of engineers. This blog post probably isn't for you. A quick google search will give you all of the very specific (and very stressful things) you can do to become a top programmer.

For the rest of people, who mostly ask these questions to well-meaning Senior and above level engineers, they will get a litany of responses.

Things like:

Go deep on algorithms because you'll use them everywhere in your career.

--or--

They might include recommending learning something like Go, Rust, or C++ because either it's a low level language or because the concepts practiced within it are carried through the rest of programming languages the world over.

--or--

The might recommend you go deep on a single language. Maybe become awesome at something like Typescript because you can learn and play around on the front end or the backend.

Or, maybe they'll say

go wide. Dabble in a bunch of different things. Becoming well rounded is important.

Personally, I find this advice incredibly stressful and not necessarily all that useful to someone who is just starting out and not entirely sure where the arc of their career is going to take them. There the type of things that make a lot of sense, as that Stanford address says, you can only connect the dots backwards, not forwards.

General Principles

Basically what I'm saying is that while it's helpful to pick some goals (Maybe you want to ship a project, maybe you want gainful employment, both are great) you don't need to obsess over what you're doing every day to reach those goals. Being thoughtful, taking care of yourself, and evaluating how you're feeling about those goals will take you way further (or at least took me way further) than trying to explicitly script out what I wanted to learn.

Honestly, being able to do those things might have made me a better programmer. There are probably better programmers who can do those things than me. But... I'm also a reasonably successful engineer. And at some point we have to trust the fact that the people will be able to make optimal decisions as they learn more.

If you're starting out (and even if you're not starting out if you're not starting out), continuing to code and learn tomorrow (and the day after) is way more important than whatever you might crunch on today. Compounding knowledge is really a thing.

The Actual Advice

So giving forward looking advice should be less about specific objectives I hit that got me here, and should be about habits I developed (or lucked into) that were useful, regardless of what I was doing as an engineer. In my experience there are two fundamental concepts (and one bonus!) you should focus on in your software engineering career:

1. Making coding a (mostly) daily practice.
2. Finding a community of people to give you feedback.
3. Skim a variety of sources so you can sound intelligent.

Simultaneously luckily, and also the most challenging, both of the first two (and likely the third) can be provided to you by gainful employment. But, not all employers will support you on either of those things, and you can get these things without employers.

1. Make coding a (mostly) daily practice

Realistically, you should code every day (that you're not on break or on vacation) somewhere between 30 and 240 minutes (that's 4 hours). The 4 hours thing is intentional, as there's some evidence that shows you have a limited number of resources.

If you're employed, you really should try to spend close to 4 hours a day (early in your career!) coding. Ideally it should be on specific problems that you can solve (code katas or other challenges will do) but ones that stretch you. Again, an employer will really help here, but even if you don't have one, the goal isn't necessarily to learn the hot new technology, or do something fancy. I've found in my career that problems will often become the most interesting to me right as they become the most useful.

The goal is like exercise, do something that pushes you just a bit, but not so much you hurt yourself. This is in direct contrast to advice that teaches you to learn a specific language or a variety. I mostly don't like it because I find long lists (without priorities) incredibly stressful and not all that useful towards doing the thing of learning.

Resources

Early on in code bootcamp, I'd pick pretty simple UI on sites like Dribbble and try to replicate them to my best ability, or maybe do a couple of Code Katas. Though, it's really best to try to solve real problems that interest you. Doing mock work can only take you so far, as the types of problems you'll run into there are often not all that useful to what you're currently trying to accomplish.

I've tended to dislike most tutorials, but Front-end Masters generally has some of the widest depth and breadth of entry-level (and further) content that I've found.

2. Find a Community of People To Give You Feedback

This doesn't have to be a large community (more than 1 other person will do, though ideally you want a group of 4-6 people). Basically what you want are people you can talk about the problems you're facing and ideas you can bounce off of. It's even better if they'll review your code. It's the best if there's a variety of experience levels.

Honestly, the best help I ever got was being able to be around other experienced devs who were willing to give me the time to walk me through what they were doing or what I was doing wrong. The most important thing is that you're stretching yourself just enough outside your comfort zone.

Being involved in a community of people you trust will also expose you to way more new ideas (in a way less stressful way) than you could ever tackle on your own.

The Job of It All

Look, the easiest way to do this is to find a job that provides these things. It will give you the impetus to work on interesting problems and the final compensation to allow you to do so. But not all jobs are created equally.

If you're looking for a job that meets the "community" requirements, here's what I'd recommend.

1. Find Senior and above engineers.
2. It should be a place that believes in the value of having people who ask obvious questions
3. Code Reviews should be an important part of the culture
4. It should want you to ship code early and often

Job or Otherwise

So if you don't have a great job, or you don't have a job yet and want to build a community, here's what I'd look for. What I want to note here is that it's not necessarily important to look for a mentor or someone to fulfill a particular role. Your goal is to get reps and to get reps working as closely as possible with other people. Tying yourself to another person (experienced or otherwise) won't necessarily facilitate that goal.

Will Larson has a great post about why activity-oriented communities are more stable. What it comes down to is that you want to be around people who are trying to solve a similar set of problems as you (from very similar experience to much more experienced). This means you're not necessarily looking for people who you immediately get along with.

This means that a group you worked closely with at a bootcamp might not be the best bet. If you don't have an obvious community in front of you, here's what I might try to accomplish:

1. Find 2-3 other developers of your skill set who would be interested in taking a course with you. It can be a free course online or
2. Agree to go through it at the same pace and create a new codebase where you all will work. (This could be )
3. Push PRs for this to the same codebase and ask each other for critiques on your PRs.
4. Find a senior dev (or set) who can answer specific questions for you. (This could be in the form of a local slack channel or someone connected to you through your bootcamp).

The other thing to note here is that people who make good friends will often not make great teammates from a learning perspective. The stuff that makes you like hanging out with them (similar opinions, diverse interests) make them not always great for learning (you want similar interests, diverse opinions). I've made this mistake too many time to count, thinking that people I get along with will make great study buddies and sometimes they do but very often they do not.

Bonus 3. Skim a Bunch of Articles

There are a few truly great email lists I think genuinely changed my life as an engineer. Basically there were a few times when I would chat with people at the many Meetups I would try to go to desperate for a job, where I would carry a conversation longer than I had any right to simply because I had read an article and new some vocabulary.

At first I thought this was faking it, but as I've gotten more experience I've realized that having a list of stuff you can wave your hands around about, and stuff that you know enough to ask some well-timed questions will serve you really well. Most of the stuff I've never had to learn in depth, but the couple of times I've needed to dive in (I'm thinking about Serverless and Typescript most recently), having that initial vocabulary experience was really helpful in guiding me towards learning the stuff more quickly.

The scanning part is also incredibly important here. Articles will want you (beg you, in fact) to read them deeply. You simply don't need to do that unless you really want to. Getting comfortable with vocabulary and terminology is more important than trying to pick up Go or Rust from the hot new blog post.

Resources

The reality is there aren't that many awe-inspiring articles going around the internet every week. Just find one you want to graze, and go through it. Here are a couple of the general purpose ones I find the most helpful:

• Progamming Digest
• Pointer

Running a successful meeting feels great. I thought that I would try to put together some ideas about how to make your meetings more successful. One of the biggest misconceptions I see is that a successful meeting means following a set list of items. Stuff like creating an agenda, setting time limits, taking notes.

These are necessary, but not sufficient, components to meeting leadership. Meeting leadership is a skill that requires practice.

You, too, can be as successful as this obligatory stock meeting image

Before We Begin, a Caution Sign

A well-reviewed or poorly reviewed meeting has minimal relationship with its effectiveness.

Let me restate that for emphasis:

How well your meeting is reviewed has a little to do with how valuable it is for your team or your company.

There are a couple of reasons for this:

Humans are bad at judging whether they've learned something

As books like Make it Stick show, learners rate professors' content as more enjoyable and more effective when they lecture. But lectures are some of the least effective ways to pick up new information and concepts. For example, fuzzing the text on a document makes it more likely you'll remember it. This is because when it's harder for us to understand things, our personal mushbuckets (our brains) assumed it's more important.

Does this mean we should go ahead and make all meetings miserable? F*** no. But there are different ways to make meetings enjoyable and make meetings effective. Some of these techniques will help with both, but they are different outcomes.

Humans Like Being Around Other People

While engineers grumble about being in meetings, they love chatting with other people. (This can come in the form of coffee breaks, hallway meetings, or Slack conversations). So, when we talk about if a meeting was successful, we have a built-in bias towards keeping them.

Meeting Feedback Has As Much to Do with that Person's Week As It Did with that Meeting

This honestly deserves a separate article. But… basically, a meeting's length and timing has a larger impact on our perception of its value more than its content or outcomes.

A poorly run meeting on a Friday will be more enjoyable than a perfect meeting in the middle of a Wednesday.

End Caution

Even with all this being said, you should try to run good meetings. Feeling productive (and being productive) is important. And if you can run meetings that are successful and fun, your techniques will be more likely to spread.

The Overview

Here's the Tl;dr. Six skills you should try to focus on.

1. Meetings should have a point (and culture meetings work only so many times)
2. Your subject should be specific
3. Keep control of your meeting
4. Be explicit about what participation looks like
5. Use status on a team and within an organization to mimic good habits
6. Restate takeaways liberally

As a meeting facilitator, you will need to take proactive action to change other people's behavior. This could put you in disagreement with the people around, which can be uncomfortable. Often focusing on these goals means focusing less on the content of the meeting. That's ok. The goal isn't to take control for the sake of asserting control. The goal is to use each of these as a tool to help people share ideas and reach their goals.

1. Meetings should have a point

One of the first things you'll hear about good meeting leadership is to set an agenda. Before that, you should visualize what a good meeting looks like. If it's a presentation, how do you want people to respond? Should they ask a lot of questions throughout? Do you want them to do something after the meeting? (Like write code or designs) If it's a discussion, what would a successful discussion look like? Do you want to retrieve information from a specific person? Is there a stakeholder in the room who you need agreement from?

Every meeting has a point, and it's worth visualizing what that looks like, beyond the agenda.

People will tell you it's important to set all of this out in a document in advance. And you absolutely should if you can, but it's more essential you have the picture in your head before you start. And then once you start, you can compare the picture in your head to what's going on. And that gives you the ability to guide the direction of the meeting. You can choose to bring it back to your original vision or let it flow in the new direction.

Danger Zones: Pushing Information (Company Culture Meetings and Presentations)

No one likes information pushed at them. The "this could have been an email" can seem like a helpful note on a different way to communicate. That's a lie. This "could've been an email" means it's getting trashed or archived, and the time was useless.

Look, sometimes you absolutely have to make sure that information comes across. Sometimes you need to do meetings for culture or to push critically important stuff because it's the least "lossy" format you have.

But often, for meetings that "could have been an email" or that are pure "information pushing". It means that you haven't thought enough about what you want participants to get from you.

I've learned the hard way that having meetings just to keep people informed or "in touch" isn't that effective. Maybe you think you're rolling out some new technology that other engineers need to know about. Maybe you think various departments know what everyone else is up to. But giving some contextless mush is going to make people tune out.

You should spend a bunch of time trying to figure out what your participants care about and why. If you're thinking to yourself (crap this means I have a lot of work to do), yes it absolutely does. The larger the audience means the more time you're taking up, means the more time investment you should put in up front to make sure that time is well spent.

2. Your Subject Matter Should be Specific

I think we all have this belief to reach the widest audience possible we need to generalize. First Focus, Then Simplify has a great discussion on why this is such a bad idea.

For one, if you're abstract, you're going to get a bunch of abstract aphorisms back. For example, if you ask a developer what the best database is you're going to get an answer about their favorite pet database.

I would advocate for two steps towards meeting and topic participation:

1. Broadly frame the problem you are trying to solve or the thing you are trying to do.
2. Quickly use specific examples for help, support, or explanation.

This can feel incorrect. If you don't give your participants all the necessary information up front, how will they know how to participate? In most of my experience, giving participants every piece of info, means it's framed in a way that I find is useful, and not what they find understandable. Jumping to examples, gives meeting (and presentation) members, the chance to ask questions. And those questions will help them understand your issue better than how you could have explained it up front.

There's some amount of "feel" to this, knowing how much is too much or not enough. I would err on the side of providing people less information and letting them ask it rather than overloading them. I promise it won't make you look stupid and will actually get participants engaged in the process.

3. Keep control of your meeting

No one wants to be the asshole that tramples over other people's thoughts or time. Especially not publicly. But the opposite is a meeting that loses the thread or runs out of time.

Acting as a facilitator means keeping an eye on how long people have been talking, considering whether their topic is relevant to your goals, and taking proactive action to correct this as necessary. This can include interrupting people, even mid-argument. It also means keeping an eye out for people who haven't spoken, reading facial features (or intent in texts). I've found saying something like "Hey your tone is making me think you have doubts, is that accurate?"

While time is an important component of this, it's the first step. A quick statement that doesn't contribute to the meeting goal should be scrutinized. It either means your goal needs to change (which is ok!!) or it means you need to discuss it elsewhere.

This sort of facilitation is also helps improve equity within the conversation. Keeping control means you can pause for questions and to restate critical points.

What to Avoid: Cursed Conversations

Some people call it swirl, others call it circling, I call them cursed conversations. These are moments in meetings that are high emotion and low outcome. Where everyone feels attached to what happens but little control over changing it.

Ironically, meeting schedules are a perfect example of cursed conversations. People often have slightly different schedules which makes coming to a group consensus hard, and they feel emotionally attached which makes compromise feel impossible.

My recommendation here is to find someone to make a decision and move on. It can feel rough, but cursed conversations are best resolved by ownership and experimentation, not consensus.

4. Be explicit about what participation looks like

For most people, meetings feel most effective when the narrative of the meeting feels like they got from point A to point B emotionally.

This can look like one of the following: a. They learned something they can directly apply to their life. b. They learned something that adds to their knowledge about something they care about. c. They felt like they were able to meaningfully contribute. d. They got to spend time getting to know the people they care about and thereby enriching your life.

Your job is to make sure that your participants… uh… participate in a way that helps them get to one of those goals. (While still meeting your objectives).

To that end, it's helpful to plan out what type of participation you think your meeting (and goal) fits best. Then, you can work backwards to create an agenda that gets you to those points. This can also give you a signal for guiding a meeting productively. If people are trying to meaningfully contribute where they only have actionable takeaways it tells you something. Maybe you need to frame the conversations, so they have more ownership of what's happening. Maybe you need to give up more control. Or maybe, you need to pull back and clarify that this is a decision that's been made, and you're looking for engagement through feedback about the results.

Meetings Are Culture

One of the weirdest parts of online business culture (i.e. LinkedIn) is the concept that Company Culture is sacred. But also, there are a million how toes on the best way to copy someone who interned for Jeff Bezos once and wrote a medium article about it. In my opinion, meetings are a primary way that company culture gets defined. So if you care about your company culture you should really care about your meetings.

If you want your culture to be loose and friendly, you should be able to envision people cracking the occasional joke without getting off topic. If you expect your company culture to be formal, efficient, and always on topic, you can be prepared for that too.

5. Use Status To Set Good Examples

One mistake I see get made a lot is by people who kick of meetings and who want broad involvement. So, they throw out fun words about how everybody is invited, and it’s open season for anyone to contribute. And then shocker you get a lot of blank faces and not much involvement.

While it makes us uncomfortable in tech, the reality is the manager, or Senior, or Staff next to our titles means something. It means that people will have a different way of thinking about your behavior. And we can use that to our advantage. High status individuals can set the standard for behavior we'd like to see out of a meeting.

I don't think we talk enough about the fact that once you're in an "important" role, there's a lot of pressure on you to continue to appear "important". This can take the form of behavior where high status individuals only contribute to things comfortably in their domain. It also can take the form of not asking questions that might appear novice.

I think leaders should take more time to carefully and thoughtfully display the sorts of risk taking behavior they would like people to see. Think Again by Adam Grant differentiates two types of confidence: social and knowledge-based confidence. Often humans tend conflate the two, but they are very separate. Someone can show low knowledge confidence while retaining high social status. They do so by signaling their social comfort in the situation while being open about their gaps of knowledge or uncertainty.

This can also take the form of inviting disagreement. For a large portion of meetings disagreement is a key form of productive behavior to gather the best ideas and also create unity. When leaders speak it can create a defacto "right answer". As Being Wrong discusses, one way to achieve this is through "self-subversive" thought. Basically, you couch your statements, in "I think" or "Maybe" or "In this context".

The irony is that most writing on the internet is the opposite of this. We're taught to go forth boldly and say what we mean lest other people not take us seriously. Especially in meetings and group settings, leaders should do this more to make sure other people feel comfortable contributing.

6. Restate takeaways liberally

I think people occasionally have a fear of repeating themselves. We tend to associate repetition with age and the mistakes that come with losing our faculties to time. But… the reality is, participants will not understand what they're learning the first time, or the second time, or the third time. This means when key points and action items pop up you should repeat them. Repeat them so everyone knows what's happening and then repeat them at the end, so we all know what we learned.

You should absolutely throw the brakes on a meeting where you aren't 100% certain everyone is on the same page. And this is true if even if you aren't leading the meeting.

Lemme repeat that: if you have sincere concerns that people aren't all on the same pages you should feel empowered to slam the brakes. Jumping in to say, "Hey, that felt important, and I'm not sure we're on the same page, here's what I heard [restate what you heard]" can make sure everyone is hearing the same things from a conversation.

I would recommend using a template for this:

[X Person] is going to do [Y Thing] by [Z Time]

-- or --

We all agree that [X Project] is going to use [Y approach] instead of [Z Approach]

Say this when you first come to the conclusion, and then again at the end of the meeting. At a minimum.

Bonus: Trust Your Participants... Seriously... Trust Them... It Will Be Ok

A lot of modern meeting design focuses on pushing information to people who are aggressively disinterested in what you have to say. As though the main goal of getting together with other people is that they should have memory retrieval of a specific bullet point or phrase that you said.

That’s frankly stupid. You need to put in the effort to make sure that people you invite to your meetings can be successful participants. If you can't think clearly about why someone wants to be in your meeting, it's a decent signal that they shouldn't be there in the first place. Making your content simpler and creating an agenda isn't going to help if you don't know why everyone is there.

So let me just say this:

You work with a bunch of smart and thoughtful people.

Lemme repeat that again

You work with a bunch of smart and thoughtful people.

Create a vision of the type of meeting you want to have. Build a picture in your mind of how you want people to participate in the meeting. Use your status to promote that behavior you want to see. Learn from the meeting and improve for the next time.

It's a genuinely hard problem to solve, but it doesn't require tricks, or feints of hand, or sleights. Once you manage that you can have fun meetings that are also productive.

Dan Abramov made a great addition to this in Optimized for Change. When you design an API, it's not good enough for you to make a system that's good now. Your system has to be good through time as well. Well, I'd like to propose a 0th order concern you'd have.

When you’re talking about a brand-new framework, pits of success are an important way to help push your users into beneficial patterns. I think, generally, new frameworks have done a good job of this. Elements that jump out are React’s thoughtfulness around functional components, Rust and memory safety, Go and concurrency, Typescript and type safety…

But these are all net new systems. What happens when we add things to old systems?

Outfit Your Explorers

Most systemic change doesn't start fresh. It starts in existing systems that have lots of gnarled roots and stuff that can easily trip you up. Most of your users are tired of just trying to make it through the world and not thinking about what mountain they're going to have to climb (system changes, package or version upgrades, coding language changes).

Yes, it would be great if your users were on that shiny new system where you could gently guide them from decision to decision. But they're not. It's way more likely they're in the 99% on an old version of some framework trying to make their project successful.

And in those cases, pits of success make about as much sense as… well… wanting to fall in a pit while scaling a mountain.

This pit of success can be yours for the low low price of…

Waiting for a pit of success means sitting around wondering how to use the system we already have and what it means to get there. It’s less about falling into (or waiting around for) best practices, and more about traversal. Waiting for a pit of success to be built means waiting around for a standard that might never make it to you, when you likely needed to ship your product yesterday.

In these cases, I would like to propose another set of metaphors, related more towards expeditions. Things like:

• equipping your explorers (documentation of teams who’ve encountered previous migrations)
• Separate parties (allowing for different groups to do a reasonable amount of duplication to understand what works best)
• Recovery mechanisms (how do we roll back changes that may not have worked)

Standards, best practices, and the like work really well on flat surfaces, and are absolutely things we should put into practice when creating brand-new frameworks and functionality. But as systems designers and leaders, we also need to figure out how to help our teams travers the mountains and get to the shiny new functionality we’re hoping for.

General Hypotheses

Serverless is a huge pain in the ass right now. On top of the fact that it’s an entirely new way of thinking about things, there aren’t the same standards you can lean on. That being said, with the money being invested in serverless, the types of things that feel hard now won’t feel as hard in the future.

While in the short term, the learning curve for these pieces of functionality feels even more arcane than code, in the long term, as time and money gets invested, these pieces of functionality will be cheaper and easier to configure and manage. Why bother having to configure your own “worker queue” when you can click a few buttons in AWS and ship a single file function to manage it for you?

But how do we get there?

Why People Are Actually Anti-Lambda

When people talk about serverless, they tend to bring up a number of very specific concerns like cold starts, the inability to run for more than a limited period of time, or weird issues like database connection pooling. I believe these are proximal causes for what is actually a fundamental transition for how we think about software engineering.

I believe Swyx’s post about the end of localhost. Basically, as systems get increasingly complex (and we’re honestly almost already there), less and less of our system development will happen on a personal machine and more and more development will happen “in the cloud”. For a group of people who have made their bones on having a lot of control over their set-up and environment, moving to patterns that are increasingly defined by vendors (and vendor lock in) feels scary.

As a backend developer, a lot of what we thought about when it came to things like “building a system”, it meant creating a series of interrelated entities (structured by foreign keys) that encapsulated a strongly articulated set of ideas. In the serverless world, a lot of these concepts disappear entirely as things like “Snowflake” do the data management, and separate entities live in separate databases entirely.

I’m in agreement with people in the sense that sometimes lambdas really are overkill. But, I don’t agree that “servers accomplish exactly what lambdas can”. From a purely technical perspective, it might actually be true that a high performing monolith is cheaper and more available than a set of lambdas. However, organizationally monoliths introduce a huge number of problems related to deployment and maintenance that creates chokepoints that are hard to untangle.

Knowledge I’m coming in with

Honestly, I’m in agreement about this skepticism and concern. However, regardless of if it’s the right choice, it’s certainly the choice we’re headed towards, and it’s worth understanding what’s going on under the hood. I have enough experience with books like Accelerate to understand the general concept of why lambdas are good.

My theories are twofold:

1. Devops resources are the scarcest in the world. Building out EKS clusters is a pain in the ass, and trying to maintain them is harder. “Serverless” technologies basically mean you have to spend less time on that stuff as a business.
2. While single servers are great and scale wonderfully, they actually create chokepoints on the development side because the more developers who have to commit to a single infrastructure, the harder it gets to actually make changes.

I’ve committed code to a repo that is a lambda, and I’ve done some various kicking around on things like AWS SAM and Serverless. What I don’t have is the habits to know what tools to reach in with as new things pop up.

Goals for this Experiment

Have a deployed, publicly accessible API that operates similarly to an existing API I have deployed previously. Likely this one: https://github.com/play-this-tonight/games-api

You’ll notice that right now this is only a Graphql server. To get the full benefit of all of this, I’m likely going to play around with REST concepts as well. The goal isn’t to create a server that is maximally “correct” but a server that uses different concepts to understand their benefits and use cases.

I also want to end this with a list of answers for basic questions I have about how I can use different serverless concepts.

TL;DR Goals

1. Deployed serverless “server”
2. Answers to my list of questions about serverless

My Approach

I’m going to ask a question, or try to accomplish a basic task that I already know how to do. This might look something like “How do I use DynamoDB?” or “How do I run things locally?”. I will write a corresponding blog post and create a GitHub branch that encapsulates these learnings, sharing both out. This roughly follow’s Julia Evans’ advice on Learning Skills. Find an area of my understanding that I don’t know in serverless, compare it to something I do know, and research until I can understand the differences.

I plan to follow Jessica Kerr’s advice to keep projects separate. By splitting them up into as small of concepts as possible, I can attempt an implementation, get some basic learnings, and move on quickly without getting bogged down in complex and interwoven concepts.

More Practical Concepts

I'll be using AWS as the service provider, assuming many of the concepts will be transferrable, and I will lean into free and cheap sources wherever possible. This might mean some concepts (like a postgres database) go by the wayside in the effort to keep control of costs.

I'll be building using Typescript and Node because that's where I'm most comfortable. I want to focus on system building not language learning.

Let’s Go

And with that, we’re off. Time to go learn how to lambda.

I’m picking the serverless package because I’ve taken the Serverless Course and found it the most easy to deal with.

The GitHub repo can be found here: https://github.com/Stuwert/cipher-serverless/tree/local-development

Initial Installation

0. Install the serverless package

   yarn add global serverless

You could also run this with something like npx instead of globally installing it, for setup. (That would basically look like prepending npx to the start of any serverless or sls commands.

1. Create the directory

mkdir cipher
cd cipher

2. Create the template

 sls create --template aws-nodejs-typescript

3. Init git repo

 git init

4. Install our code

yarn install

Ok, so let’s see what we’ve created.

Some initial thoughts jump out to me. For one, the config file is a serverles.ts file, not serverless.yaml. While it might not matter much between the two, it’s a bit of a divergence from what something like AWS-SAM creates. I don’t mind reading a TS file, but was a bit caught off guard.

Other things jump out, like the fact that there’s only one item in the "scripts" section of the package.json is a "test" command that errors.

The big thing that jumps out is that this doesn’t really look like a server I’ve ever seen. A lot of the concepts that I might understand just don’t appear.

For example, route definitions don’t live in code, they live in configuration. src/ is broken up into functions/ and libs/. I’m definitely used to seeing more directories floating around.

Another question popped up:

Where would I put my ORM, if I wanted to use one? Should I not even think about using one? What if I want to share an ORM across functions?

Booting it Up

Before I jumped into the codebase, I’m going to make a couple of changes first:

1. Install serverless package within the repo directly
   1. This is so, moving forward, the version of serverless we run in that directory will be pegged to the directory and not to the system at large.
2. Run yarn sls --help, so I can understand what’s going on under the hood.

I find myself running the --help flag on a lot of command line functions to get a sense of how well documented they are, and to see if there are any helpful tips and tricks. This is the first half of what serverless spits out.

    Options
    --help / -h                     Show this message
    --version / -v                  Show version info
    --verbose                       Show verbose logs
    --debug                         Namespace of debug logs to expose (use "*" to display all)

    Main commands
    deploy                          Deploy a Serverless service
    deploy function                 Deploy a single function from the service
    info                            Display information about the service
    invoke                          Invoke a deployed function
    invoke local                    Invoke function locally
    logs                            Output the logs of a deployed function

The first thing I notice is invoke local, so I’ll start there. I ran yarn sls invoke localwhich gave me an error: Serverless command "invoke local" requires "--function" option. Run "serverless invoke local --help" for more info I ran the --help flag again to see if I could glean anything else. Invoking help told me that the function flag is missing (i.e., it doesn’t know what function I’m invoking). I did some more digging and look in the serverless.ts file to discover that hello is the function name.

So I ran the command

yarn sls invoke local --function hello

This gave me another error:

    {
        "errorMessage": "Cannot read properties of undefined (reading 'name')",
        "errorType": "TypeError",
        "stackTrace": [
            "TypeError: Cannot read properties of undefined (reading 'name')",
            "    at hello (cipher/src/functions/hello/handler.ts:9:34)",
            "    at runRequest (/cipher/node_modules/@middy/core/index.js:86:32)",
            "    at processTicksAndRejections (node:internal/process/task_queues:96:5)"
        ]
    }

This looks like the type of code function that I could understand. It seems like a function was invoked and failed because it didn’t have the data it needed to invoke.

This looks like a pretty standard error. It seems like the event being invoked needs something passed to it, which makes sense to me given that it’s an API lambda that’s expecting some incoming data to process.

So… how do I figure out how to start a local server with serverless?

Now to the Google machine

Searching for: api gateway serverless local The first result I find is about AWS SAM, but the second result is about Serverless Offline It seems like serverless offline offers the sort of long running instance that we might expect from a server, whereas a direct invoke is a one off development pattern.

Installation is pretty straightforward

yarn add -D serverless-offline

And then it looks like it’s just a matter of figuring out where to put it in the configuration file. Plugins oughta do it.

    // Before
      plugins: ["serverless-esbuild"],

    // After
      plugins: ["serverless-esbuild", "serverless-offline"],

Then all I need to do is run yarn sls offline

This spits out the following:

➜ cipher git:(local-development) ✗ yarn sls offline
yarn run v1.22.17
Starting Offline at stage dev (us-east-1)
Offline [http for lambda] listening on http://localhost:3002
Function names exposed for local invocation by aws-sdk: \* hello: cipher-dev-hello
┌─────────────────────────────────────────────────────────────────────────┐
│ │
│ POST | http://localhost:3000/dev/hello │
│ POST | http://localhost:3000/2015-03-31/functions/hello/invocations │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Server ready: http://localhost:3000 🚀

What’s really neat about this is I can either access it through direct invocation cipher-dev-hello or via a post, request. I’m going to try both. Once with AWS cli and once with postman.

The schema generated seems pretty straightforward: export default { type: "object", properties: { name: { type: 'string' } }, required: ['name'] } as const;

Looks like all that’s required is a name.

The first thing I tried didn’t give me anything useful back. I threw a console.log statement into the function and re-ran it. (Somewhat purposefully naively). I noticed here that it doesn’t do auto-reloading. I.e., it’s not including more recent changes I’ve made. I went back to the --help function and found the --reloadHandler. This appears to mean after every request it will refresh the context and a new request will use the changes made to any of the files.

    yarn sls offline --reloadHandler

Great! That worked.

Using AWS invocation

Now let’s try with aws …

I have a bit of experience with the AWS cli. If you’re on a Mac (especially an M1), like I am, I’d recommend installing it using brew install awscli, but amazon has more detailed instructions here if you need them.

Running aws lambda invoke help gave me a pretty clear sense of what I need to do to invoke it (I’m not going to copy here because it’s going to be impossibly long). You can find the reference at this link.

This is basically where I started aws lambda invoke --function-name cipher-dev-hello --out json --endpoint-url http://localhost:3002 out.json

There are some keys here:

--out json This is the type of file being output

--function-name cipher-dev-hello is how you tell AWS what function to invoke

--endpoint-url http://localhost:3002 otherwise the CLI is going to try to invoke a function in your AWS instance.

That last parameter out.json basically tells us where the output file goes to view the results of the invocation (I’ve committed one of the successful results to git, though I think in most cases you’d have this gitignored.

And… the output is basically what we saw at the start above. (Which is great!)

{ "errorMessage": "Cannot read properties of undefined (reading 'name')", "errorType": "Error", "trace": [ "cipher/src/functions/hello/handler.ts:13", " message: Hello ${event.body.name}, welcome to the exciting Serverless world!,", " ^", "", "TypeError: Cannot read properties of undefined (reading 'name')", " at null.hello (cipher/src/functions/hello/handler.ts:13:34)", " at null.runRequest (/cipher/node_modules/@middy/core/index.js:86:32)", " at async MessagePort. (cipher/node_modules/serverless-offline/src/lambda/handler-runner/worker-thread-runner/workerThreadHelper.js:24:14)" ] }

This seems like the same deal as last time. I needed to input something to get this to work:

aws lambda invoke --function-name cipher-dev-hello --out json --endpoint-url http://localhost:3002 --payload '{ "name": "Stuart" }' out.json

But then I got a new error! Invalid base64

I did some googling and got here: https://bobbyhadz.com/blog/aws-cli-invalid-base64-lambda-error

--cli-binary-format raw-in-base64-out

The long and the short of this one appears to be that lambdas expect very specific types of invocation, and if you’re doing raw text, you need to let them know, so they process stuff for you first.

After that… I got a new error for undefined…

I added console.log(event); to my invocation function, and noticed something that wasn’t at all what I expected:

{ name: 'Stuart' }

Previously (not shown here), I found an entire event body on this invocation, but here I was only seeing the simply JSON object I had passed. To me, this meant that in the direct invoke context, there’s some processing that doesn’t happen when it’s called as an API.

New Question: Is there any way to check against direct invocation? And maybe prevent a lambda from being direct invoked? (I’m guessing no)

But the solution here is simple enough, updating the parameter of the input to include a "body" key. It got complex enough that I added it to a new file. input.json

{
  "body": {
    "name": "Stuart"
  }
}

With that update, my new invocation looked like:

aws lambda invoke --function-name cipher-dev-hello --out json --endpoint-url http://localhost:3002 --payload file://./input.json --cli-binary-format raw-in-base64-out out.json

And the output??

{
  "statusCode": 200,
  "body": "{\"message\":\"Hello Stuart, welcome to the exciting Serverless world!\",\"event\":{\"body\":{\"name\":\"Stuart\"}}}"
}

Exactly what I expected.

Great!

Invoking Using SLS

As I proofread this, I realized there was a third way I could invoke the function. And that would be directly with sls

    yarn sls invoke local --function hello --path ./input.json

I wanted to test this pattern because while it errored early on, it's probably the closest to the "default invocation pattern" for lambdas, so understanding its use seemed key.

Why All of These Different Paths?

There are a bunch of different contexts to invoke a lambda, right? Depending on what the usage that a lambda happens in, it can be useful to know how to execute it in different ways:

• If the Lambda is responding to events that will occur on your local machine, knowing how to get it running, and then hook it up to those events seem useful
  • This is kinda like what a Bull Queue would do on a long-running Node Server.
• If a Lambda is more of a traditional API, you’ll probably want to develop a frontend application against it, so knowing how to use it as an API locally is important.
  • This is your traditional Node Server.
• If a Lambda exists to process data, maybe without much external output, you might just want to invoke it separately.
  • Maybe this is a one-off function like firing something to the Event Bus or running a migration!

Knowing how to invoke each condition gives me a better feel for the different use case and a feel for how it relates to a traditional server.

One Final Step

Now that I’ve got everything running locally, I’m going to make one change to package.json I added some scripts to our package.json to easily run this handler in the future. "dev": "yarn sls offline --reloadHandler", "invoke:hello": "yarn sls invoke local --function hello"

Summary of What We’ve Accomplished Here Today

• Created a new serverless repo with was typescript
• Added the serverless-offline plugin to allow local invocation
• Invoked the function in two ways
  • Via API
  • Via Direct Invocation.
  • Via Running the handler and calling an Invocation.
• Updated package.json for easy re-running.

Summary of What I Learned

• Spinning up a new repo for serverless is pretty straightforward and painless. Things like the API gateway and a local development environment are set up for you.
  • This includes things like a basic tsconfig.json
• Direct invocation of a lambda locally is different from a post route because gateway (and the local gateway instance) does some processing we otherwise wouldn’t understand
• Serverless uses .esbuild by default for the aws typescript package.
• There was a ton of code generated that I don’t totally understand, and intend to dig into further.

As a follow up to a previous post about what to do next as a junior out of bootcamp, I got asked the question about the value of "working in public". There’s a lot of information floating around about the benefits to “work in public”, to keep a public record of the code you write and your thoughts to share it with the world. It can also mean working on a project with a publicly accessible repo, like an open-source project, or a project that other people can access.

The "internet audience"

I remember leaving my coding bootcamp bootcamp I felt a lot of pressure around the benefits of “learning in public”, to do things like keeping up my GitHub commit streaks, posting on medium (or now Substack I guess, is there DevTok?).

Having been at 3 development jobs since, I want to reflect on how I feel about the pros and cons around working in public. Is all of this a potential way to get a job? Maybe. Is it necessary to work in public?

My short answer is no.

The longer answer is below.

1. What is working in public?
2. Why is it such a big deal in tech?
3. What are the downsides?
4. What are some other options?
5. When should you work in public?

What is Working in Public?

Working in public is the idea that you should put on display you’re doing, learning, and thinking about. This typically comes in the form of a blog where you provide links to things like projects, GitHub, and examples of the stuff that you’re doing. It can also come in the form of giving presentations at conferences, or contributing to an open-source repository. This blog (yes, I did just link to my own blog), and my serverless series, are a great example of “working in public”. I’m trying to share what I learn, as I learn it, so other people can take advantage of the growth I experience.

Why is it such a big deal?

Swyx’s free chapter of The Coding Career Handbook goes into solid depth on the primary arguments for working in public. It’s well written and worth a read, but I’m uncertain if it’s the only or even the primary way that most software engineers are employed. It’s one specific way to get to a specific set of outcomes that you might want.

But, the primary benefits for you and your career, are roughly:

1. Improving Your Coding Skills
2. Having an easier time getting jobs
3. Having a higher net worth overall

I think there’s also something of a tech aesthetic going on here around being part of open source, and displaying your status by showing your dirty laundry. While I actually think it’s not a bad thing to have sharing be a status symbol, I also think it can go overboard occasionally.

Improving Your Coding Skills

Having to explain yourself to other people is proven to be a good way to sharpen your understanding and clarify places where you lack knowledge. (There’s this thing human brains do where we will assume we know something if we know someone who knows about that thing. For example, if you ask most people if they know how a toilet works, they will say yes. If you ask most people how to describe the functioning of a toilet, they will struggle).

Working in public also gives you the ability to receive feedback about what you do and don’t know. You might get new ideas from people you share the work with, or you might have someone share something you were wrong about.

Having an Easier Time Getting Jobs

Having publicly available material you can point to in job interviews can be a benefit. It can mean sharing your website for your prospective-employer to get to know you better. And, if you have some level of notoriety, maybe they’ve already read your work and have a pretty good sense of who you are.

It’s also a good way of verifying your skill. You don’t have to go through thirteen rounds of a front end interview to prove your coding chops if you’ve built some clever UI elements in a personal website, or made a significant contribution to an open-source repository. Talking about the code you’ve written at other jobs can be challenging because your interviewers are taking it on faith that you were actually doing the things you say you were.

Having a Higher Net Worth Overall

Swyx calls this expanding your luck surface area. The basic thought here is by engaging in effective marketing, you’re making it more likely to get the sorts of wild financial outcomes that we saw during the dot-com boom. This can be in the form of exposure to a small, hot startup that can exit with an outsized valuation, or getting a high-value position at a well-known FAANG*. To be honest, I suspect that this is the case, but it’s not necessarily a need.

*(Facebook Apple Amazon Netflix Google)

Downsides to working in public

I’m skeptical that there’s a specific, obvious reward for junior engineers to having a website where their work is visible. I can remember each time I’ve gone into a new job search thinking “it would be great if I had a blog” and then completely forgetting about it the second I actually had a job.

What are the downsides to working in public?

1. It takes a ton of time and effort
2. Building a small following and portfolio is a long-term investment
3. It’s probably not the primary way you’re going to find your next job

It takes a ton of time and effort

Part of being in a 40-year career means doing things that increase your longevity, not necessarily things that maximize output and keep you in the game. Working in public in an effort to get a job might not serve you all that well.

Saying “work in public” sounds great, evoking this concept of just taking all the stuff you already do and putting it on a website. But that’s probably not going to get the exposure you want (or need?). And I’m not even talking about the potentially immense emotional effort and risk involved. I just mean, when you learn something, it actually takes a lot of effort and refinement to structure that in a way that’s legible for other people.

It’s often not enough to just have a repository, or a deployed site, you likely want a solid README, or a post-mortem, or a demo or some documentation explaining what’s going on. For example, even something like my “serverless” project, can seem excessively straightforward, trying to figure out the beginning and end of each article is… an incredible challenge, actually. I’ve put a lot of thought into what order articles should go in, and whether some even belong, long before I actually click the publish button.

Build even a small following is a long-term investment

Writing can definitely pay off, but in terms of increasing your luck surface area, it’s probably going to be 1 or more years before you see the sort of cumulative impact from a blog that you might expect. I’m imagining outcomes like getting a job opportunity, getting a promotion or a raise, or making a connection with someone that might not have known you otherwise.

And, as a long-term investment, it’s the type of thing that you will only feel comfortable doing if you have the time and space to do it. If you don’t. If you have people to take care of, other demands on your time, or even other interests, you don’t need to do it. It can be an area you decide not to invest in, so you can invest in something else important to you.

It’s probably not the primary way you’re going to find your next job

Working in public is part of a constellation of stuff that will define you and your potential value to (and interest in) a future employer. Working in public can create the type of web of connections that makes you valuable. But, the actual act of being public is just a small component of that, and there are other ways to create similar sorts of value in less public ways.

Other Options: Private and Semi-Private Work

This is actually something Swyx goes into great detail in Section 1.8 of marketing yourself to your fellow humans. And, while I don’t necessarily agree with the framing, I do think the underlying logic is solid. It’s worth it to spend some effort making your work visible to the people around you. They probably know less about what you’re doing than you might think.

As an alternate to “working in public”, I’m going to focus on a couple of things:

1. Practice writing. Journal.
2. Keep a brag book.
3. Maintain a list of connections you want to maintain

Journal

Yeah, I said it. I'm old-fashioned. But other people don’t need to read your writing for it to be useful. You get a lot just by putting your thoughts out there. If keeping a blog helps your journal, great, do that. But the main goal is to write and articulate your thoughts on paper, and see where you do and don’t have gaps. It’s like 80% of the benefit of writing in public but without having to worry about things like grammar, spell check or legibility. And if some day in the future you want to turn it into a piece of writing, it’s a great starting point.

Capturing Your Output

Along with journaling, you should keep a running track of stuff you’ve accomplished. Some work you can copy screenshots, some work you might want to write a quick paragraph.

Firstly, it’ll make you feel good about yourself. While your friends might not understand that creating a login page is a considerable undertaking (as my personal experience at code bootcamp can attest), your brag book will never let you down.

Secondly, it builds a good habit of practicing talking about stuff you’ve done. Lots of the time in situations like hiring interviews, potential promotion discussions, or chatting with another developer at a meetup or a conference, you’ll want to be able to talk about what you’ve done succinctly. Keeping a brag book will prepare you, without making any of that preparation public.

Brag books also help you track your accomplishments, and build a narrative about your growth in your career. It’s surprising how many small details we can forget when we don’t record them, so having them recorded is a great way to make sure they don’t get lost to time.

In my opinion, brag documents are one of the highest leverage things you can do to help advance your career (and your personal happiness at your career).

Cultivating Your Private Network

Once you get in a job, one of the best things you can do is to cultivate an internal network of people who you would like to stay in touch with over time. This doesn’t mean being weird or conniving. All you need to do is if there’s someone you work with who you find interesting or inspiring, find some time to grab coffee or lunch with them, and make a connection. Become friends. These are the people who will vouch for you, refer you, and help you find your next opportunity. (And you for them)

Having a group of people who believe in me (and who I believe in), has helped open up potential job opportunities and given me perspective on what decisions I might make in the future. It’s very private work (and honestly people are fun to be around, so stay in touch with the ones you like!) but it’s been a huge boost to my career. And for people who are just out of bootcamp, your bootcamp-mates and your teachers can serve as the same sort of network. Keep in touch with them!

This isn't to say it's easy. Of all of the recommendations I'm making here, it's probably the hardest one. I have a set-up on my task list where every 2-3 months I'll get reminded to reach out for coffee/lunch. And I still feel overwhelmed and uncertain when that task comes around. But for the effort of coordination and keeping up, it's the career benefit and the intrinsic value of being connected to people I enjoy, appreciate, and learn from that makes it worth it.

When To Work In Public

The goal of my writing here isn’t to convince you not to write or try to commit to an open-source project. Rather, I’m trying to destress the situation where you feel like you have to do these things to have a sustainable career. If you want to write, if you would like to contribute to open source, you should absolutely do it! Just do it for intrinsic reasons and not because you feel like you have to as some sort of stress response to the scariness of the tech world.

The Exception

There is one incredibly compelling reason to work in public, and that’s in the case where you want to do the type of writing or public stuff that helps bring something to life that wouldn’t exist without your contribution. Basically, if you feel compelled to write/contribute, you should do it! It’s reasonably likely someone will find a lot of benefit from your perspective and experience. But do it because you feel intrinsic motivation from the work, not solely to try to min-max your career.

This doesn’t have to be huge, but it’s a really compelling reason to work in public and if you find one of those areas, there’s a chance you’ve hit on something really fun and exciting. (But don’t feel like you have to!)

I’m thinking of blogs like:

• https://jvns.ca/ work on answering questions about how coding and computers work
• https://lethain.com/ work on tech leadership
• https://charity.wtf/ work on tech careers

The point here isn’t their fame, it’s that they’ve found areas of interest where they felt they could make a meaningful contribution, and so there was benefit to working in public. If you can find one of those (even if it’s just for your 5 friends), you’ll likely feel the type of dedication and commitment to keeping it up as a practice.

If not, that’s ok too! You can write and work on your side projects off and on as you have the energy and time.

There Is No One Way to Shape a Career

Starting out is the toughest part of your career, especially in software engineering, where the tools change constantly, and it can feel like you’re swimming into a wide and deep ocean. There’s no one way to do it. Just keep swimming.

I’m going to cheat a bit and reorder it to focus on the stuff that I have more confidence with first.

• Serverless Framework
• AWS SAM
• AWS CDK
• Terraform

What are the options?

SAM, or Serverless Application Model, is AWS’ toolkit to basically create deployment templates for common deployment steps, along with local development tooling, similar to the Serverless Framework. Serverlessland doesn’t have any templates for TS and SAM, but sam init had a hello world example, that was a good starting point.

CDK, or Cloud Development Kit, is basically a set of packages, exposed to various libraries, that allows for developers to instrument cloud formation functions within code. Basically all that stuff you see in serverless framework’s template.ts but instead of config, it’s code.

Finally, Terraform. I honestly didn’t even realize terraform was an option here because it felt like using a jackhammer to break up peppermint bark. But the option is there, so I’m going to try it out. Similar to SAM, Serverlessland doesn’t have any templates for Typescript and Terraform together.

Serverless

If you want to know how to set up serverless locally, look here: https://www.urback.net/posts/how-do-i-get-a-serverless-function-running-locally/

AWS SAM

AWS SAM is AWS’ version of Serverless. You can find installation instructions here. I used brew on my Mac, and then ran sam init to kick things off.

:repo

:initialization

:directory

:template file

Things That Jumped Out to Me

1. The main commands to execute are sam build (builds your package) sam local invoke (invokes locally), but you can also use sam local start-lambda (or start-api) to create a running implementation that you can invoke against dynamically.
2. The output of build creates an .aws-sam directory, which is what gets referenced locally (and what, I assume, gets deployed as well).
3. Out of the box, SAM doesn’t support hot reloading.
   1. To be fair, neither does Serverless, but it’s a two line installation to get it functioning.
   2. There are multiple support requests for this with SAM, but nothing appears to be rapidly forthcoming.
4. SAM doesn’t (at least not obviously to me) have a system of plugins I can easily add various sets of functionality.
5. SAM has many of the same local instrumentation layers that serverless does, creating a local gateway that I can execute against, or creating an execution instance for the lambda to be invoked other ways.
6. SAM definitely seems more oriented towards a “pure” implementation of a lambda, i.e., a distributed set of components that are invoked via events, not as a coherent system I met need to run and develop against on my local.
   1. And… from Amazon’s perspective… I get it? But I don’t think it’s all that helpful, especially not in the situations where I’m trying to transition away from a monolith, but a lot of my functionality is going to have to come through direct invocation.
7. Out of the box, the typescript implementation has each subdirectory install separate packages, rather than having node run at the top level. This makes me think they're nudging towards a mono-repo type implementation, potentially.

AWS CDK

AWS CDK lets you write infrastructure as code. I didn’t have much if any experience coming into this early on. The installation for CDK isn’t anything special because it’s “just another NPM package”.

:repo

:initialization

:directory

:template file

Things that Jumped Out To Me

1. Having to use SAM locally means the difference between CDK and SAM is minimal.
2. CDK seems way more heavy weight than necessary for instrumenting a basic lambda.
3. That said, CDK seems like it would be great if you wanted to orchestrate many lambdas together within a single repo implementation.
4. This is one of those cases where infrastructure as code seems harder to parse than infrastructure as a YAML file. This might seem wild, but YAML files do a much better job of encoding logic that’s abstracted consistently (and that I can read at a glance), instead of trying to read and parse code.
5. CDK has similar outputs as SAM, but with cdk prefix instead of .aws-sam.

Terraform

Ok so terraform, the big beast of the bunch. Terraform, by Hashicorp, the same company that brought you VMWare (one of the first virtual machines I used in my career) lets you write infrastructure as “code”. To be honest, I assumed that it wouldn’t be applicable in this case because I thought it would be. Installation instructions can be found here: https://www.terraform.io/downloads

To install terraform I ran:

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

:repo

:initialization

:directory

:template file

You can find my repo implementation here:

1. At first, I found Terraform’s implementation easier to read, but by the end I found it incredibly hard to figure out which reference was pointing where, even compared to the yaml files I’d seen previous.
2. One thing I really liked about terraform is splitting apart terraform plan and terraform apply.
   1. Plan will describe to you what’s going to happen (which you can lock in place by defining an output) and apply will do it for you.
3. Installing and playing around with LocalStack was a helpful experience. It basically seemed like a locally implemented version of AWS (or a subset), that I could point to as a target.
4. Getting terraform working with LocalStack required changing some of my targets to local implementations.
   1. I’m guessing you can use flags to point to which environment you’re building with, but it’s still unlike any of the others where you can simply build and target things locally without any extra work.
5. However, figuring out what the endpoint I needed to hit to test it out was a bit of a pain
   1. [http://localhost:4566/restapis/wfbyj3b6bz/test/\_user\_request\_/my\_api\_route]
   2. The wfbyj3b6bz is the api ID, test is the api gateway identifier, I’m not sure what _user_request_ is supposed to indicate, and my_api_route is the route path.
6. It seems less heavy weight than CDK though, so I’d probably pick it before that one.

What about local development?

Most of the development examples here didn’t support some version of “hot reloading”, where I could make a change to the code and see it propagated on every next request I make. I am a big fan of the “guess and check” method of software development. Where I build sets of code together by having a live instance running locally, making a change, and then running an API request to validate whether that change took effect.

The fundamental problem all of these things get into is that a lambda is a zipped package of code that gets executed and built from an S3 bucket. That’s way different than a long-running process that can be viewed and rebuilt. Is it possible? Yes, as Serverless and SAM point to, but it still ends up being a slight deviation from what the deployed code is going to look like.

Is this type of development possible in lambda land?

I suspect my answer to this, from a pure “serverless” perspective, would be to… not do it. I suspect they would argue that, like a component of a car, you don’t test the individual component by deploying the car and then running it. If I wanted to “guess and check” I should run tests in watch mode. And then, I should define clean interfaces with the components of the rest of the system. Maybe, I can run it once locally before I deploy, but that would be it.

But as I said when talking about the difference between pits of success and architecture transitions, I think there is often some conflict in those goals. The end state of building this way makes sense to me, but it’s a huge challenge in the interim: where getting things working might require something as bonkers as having a FE that talks to a monolith BE that directly invokes a lambda locally. While long term I might never need to set up the ecosystem on my local, when I move from here to there, I probably will want to, which means systems like Serverless will be way more interesting to me.

Summary and Appendix

The thing that’s jumped out to me the most through all of this, though, is how different each of these implementations seem to be. In the “traditional” server world, there might be a Go server, or an Express Server, or a Flask Server, but I could open up the GitHub repository and pretty quickly have a pretty clear sense of the implementation. Similar to serverless engineering generally, it’s an entirely different way of thinking about the world.

I’m spending way more time thinking about my infrastructure and set up and way less time thinking about code… On one hand, that’s strange and kind of boring. On the other hand, it aligns pretty closely to how the new serverless world works. Less, more lightweight code, more thoughts about the underlying architecture that goes along with it.

As the complexity shifts further down the stack, there seem to be entirely different ways of even implementing deployment and maintenance steps, so “reading” the codebase becomes its own adventure.

Ecosystems

It seems like there’s a lot of different ways to build out a lambda function, to varying degrees of success. One critique of this approach is that this is a “localhost” first look at how to spin up and maintain lambdas. This isn’t really the “serverless world” way of thinking about it.

For example, both serverless and localstack seem great, but they’re each heavily invested in node and python ecosystems. You might pick one or the other, so you don’t have to maintain python and node simultaneously in any given package. AWS SAM has similar problems in that it seems built specifically for AWS (internal) workflows… which is fine, you should do things the “framework” way, but it doesn’t seem to be updated with the same speed that other frameworks are.

This is still an area of active development though, and things like SST, built on top of CDK but with a more straightforward set of interfaces for developers to build on top of.

My basic (and boring) take is… you should pick a framework and lean into it.

As I’ve mentioned a few times, The Washington Post has put together quite a shockingly excellent video game reporting outfit. They consistently write interesting pieces that often tackle the heart of core issues like matchmaking, and their game recommendations find gems that often don’t come up in bigger lists.

10 Great Games to Play on iPhone is an example of that. Outside of maybe one exception (I’m looking at you Mario), I wholeheartedly endorse this list of games you should check out. I’ve even written about a couple likeCard of Darkness and Dicey Dungeons, two of my top-10 all time games.

Like the article, I’m going to select a variety for purchase or through either Apple Arcade or Netflix’s new gaming subscription service. While there are no free games here, I’m comfortable that you can spend $5 and be satisfied. Either through a month of Apple Arcade, reusing into your (or a friend’s) Netflix subscription*, or by picking up one of the paid games for $5, you’ll feel like it was money well spent.

* I don’t think you should buy a Netflix subscription for any of these games. $15 a month is a lot to ask for games, unless you’re going all in on a game pass subscription, or something.

1. Poinpy

It’s hard to describe Poinpy. Well, it’s hard to put it into words in a way that makes any sense at all.

You play as a green creature that’s jumping off walls trying to collect fruit for your friend the cat. You’re trying to collect the correct combination of fruit. If you land and your fruit is correct, you get to continue. If you don’t, you’ll get burnt and lose a life. And… that’s the game… kind of?

The game is a cross between an infinite runner and a platformer. You jump ever upward, collecting fruit. And when you land, you either have the right recipe or you die. The goal is to collect enough fruit to make it to the final boss battle.

Poinpy’s designer, Ojiro Fumoto, also created Downwell in 2015. At the time, it was a paid app with a punishing difficulty curve. I never made it past the second level. But it became a cult classic and a hallmark of what was possible with phone based game design. Poinpy feels like it learned from the mistakes of Downwell to create a brighter, more inviting game that has a high skill ceiling but is more available to people who aren’t ready to invest 10-20 hours into it.

Poinpy makes incredible use of the restrictions (and benefits of the phone). Its portrait mode design has the player jumping up and down, using the flinging mechanic excellently to have players create wild combos as they reach for ever greater heights. And I haven't even gotten into the incredible combos, avoiding (and jumping on) enemies, and the various worlds you hop through.

Poinpy is on my shortlist for game of the year.

2. Ordia

Just like Poinpy, Ordia is all about leaping your way up a tunnel. Unlike Poinpy, it’s not a rogue like, so there are distinct levels (often that you can complete in a minute or less) with checkpoints and save states. The art style is immaculate, and there’s enough side quests (timed and “hard mode”) that it kept me coming back even after I had dropped it for a while.

While Ordia is a real-time game (i.e., not turn based), it’s not frenetic the way that Poinpy is. You can take a breath and collect your thoughts as you jump from point to point. There are a few intense moments separated by easier patches. The clever pacing is what makes this game feel so fun to keep coming back to.

3. Railbound

Railbound is by the same designers who brought you inBento and Golf Peaks. Railbound has that same charming art style with brain bending puzzles that I found so charming in Golf Peaks.

Each puzzle takes about 1-3 minutes long, and the comic book style art is fun to look at while you do some gentle puzzle solving. Railbound puzzles fit into that delightful space where they’re not quite easy enough that you can brute force them, but not quite hard enough that you have to spend a lot of brain power trying to figure them out.

4. Please Touch the Artwork

Similar to Railbound, Please Touch the Art is a puzzle game about sequencing events in the correct order. Unlike Railbound, however, it’s about building your own little Mondrian (and other abstract) pieces of art. Most puzzle games are collections in a menu that you quietly work through. Please Touch the Artwork has a really charming conversational narrative style that’s reminiscent of Thomas Was Alone with a “narrator” that directs you to a puzzle to tackle. And, the puzzles themselves are more about brute forcing the right combination of options, rather than trying to understand a core concept.

Some games really are vibe checks more than they are about the puzzle of it all, and this is a game that represents the top of that genre on the phone.

5. Transformers: Tactical Arena

I hate Clash Royale. It’s filled to the brim with shady free to play mechanics, and the underlying game isn’t fun enough to keep coming back. I find most multiplayer iOS games to be tired retreads. So, I was surprised when Transformers: Tactical Arena wasn’t just addictive, it’s charming and fun.

I’ve written about this one previously, and I’m surprised I like it so much. Unlike the Star Wars game of a similar concept, also on Apple Arcade, I connected with Tactical Arena. For me, the big difference is how much variability in play that having “two-sided cards” (cards you can play either as a car or a robot) gives the game. It adds a lot of decision-making to the game, and each game I’ve played has felt like it goes down to the wire. It’s definitely a bit more of a stress inducing game, but I appreciate that (because it’s part of Apple Arcade), there aren’t any freemium mechanics to get in the way of my enjoyment of the game.

You can find the resulting here repo: Cipher-Serverless/Multi-Service.

Through my experience testing this out, I learned that there are potentially a few different ways to define the boundaries of different “services” within a single repo.

By that, I mean you could be talking about:

1. Chunks of code that re-use the same configuration but get deployed to different lambdas within the same cloud formation stacks.
2. Completely separate cloud formation stacks that don’t interact with one another.

Going in, I assumed that multi-services were the first, but I quickly learned that was not entirely correct.

1. Creating multiple services within a single repository
2. Orchestrating their deployment with Serverless Compose
3. Creating multiple lambdas within a single stack
4. Adding Shared Dependencies
5. Wrap-up

1. Creating Multiple Services within a Single Repository

This step was rather straightforward. I created a new directory at a top level called /players and moved the existing files into it. Then I created another directory /games and ran sls create --template aws-nodejs-typescript to generate the second set of services into it. And that was… pretty much it.

My file structure looked like:

│   ├── games
│   │   ├── serverless.ts
│   │   ├── src
│   │   ├── tsconfig.json
│   │   └── yarn.lock
│   └── players
│   ├── package.json
│   ├── serverless.ts
│   ├── src
│   ├── tsconfig.json

Having had a bit more experience with Yarn Workspaces, I knew that I needed to create a top-level package.json with the setting private because I knew that’s what define a thing as a yarn workspace.

{
      "private": true,
      "name": "Cipher",
}

I also made sure to name the new sub-packages appropriately

{
      "name": "players",
      // ... more stuff
}

And that was that.

The issue here was that to run each of these components, I’d either have to run something like: yarn workspace players run dev Deploying would be similarly onerous: yarn workspace players run sls deploy

Which would execute the following script: "dev": "yarn sls offline --reloadHandler"

I had two functional repos, but this wasn’t exactly the level of integration I was looking for. These services knew nothing about one another and would have to be deployed and maintained separately. I wanted to know if I wanted to treat the deployments like interrelated components.

2. Orchestrating Deploys with Serverless Compose

I did some googling and saw (rather recently) that Serverless introduced some new concepts related to they call multi-service deployments this year. There were a couple of surprises here that I didn’t totally understand. For one, each sub-service continues to have its own serverless.yaml or serverless.ts component. Secondly, I would need to create a serverless-compose.yaml at the top level of the directory to manage the two.

Luckily, the compose file is incredibly straightforward:

# serverless-compose.yml
    services:
      players:
        path: players

      games:
        path: games

I also learned that I would need to install a couple of packages at the top level of the directory: serverless and @serverless/compose . But from there, running yarn serverless deploy was incredibly straightforward.

And the docs also explain other ways to further interrelate the different services with one another. I was, however, surprised to notice that two separate cloud formation stacks were created. This contracted my initial assumption that one repo would correlate to a single cloud formation stack.

3. Creating Multiple Lambdas within a Single Stack

I went back to the drawing board and started digging around the serverless.ts configurations that had been generated. I realized that I could add multiple functions to a single configuration. I created a new handler called /health and required it into the games/ directory, and re-ran the script. Voilà, a second serverless lambda was created within that same CloudFormation stack.

I realized I had been working under the assumption that SLS would create a single package for each service and then deploy the same package to different lambdas. However, SLS under the hood appears to be a bit smarter (you can also add pruning to this) about what it deploys. It will create a separate zip for each function being deployed in the configuration.

It suggests to me that the purpose of creating separate “services” would be to be clear about different boundaries across the application. I’m imagining a situation where you want to distinguish between different types of configuration:

- Admin Groupings
- Events
- Step Functions
- GraphQL
- REST

But that still work around a distinct set of functions that are worth maintaining and thinking about within the same repository.

4. Adding Shared Dependencies

But the real reason you’d want to have a multi-service repository, is if there’s a grouping of packages that maintain shared dependencies across one another.

Getting this working was actually the most complicated portion of the entire work, as it required changes to both package.json files and tsconfig.json files.

I had to do some googling to try to piece together what was going on under the hood here:

• https://stackoverflow.com/questions/57679322/how-to-use-yarn-workspaces-with-typescript-and-out-folders
• https://classic.yarnpkg.com/lang/en/docs/workspaces/
• https://semaphoreci.com/blog/typescript-monorepos-with-yarn

I started by creating a new top-level directory structure with packages/ and services/. My idea here is that packages would represent shared dependencies, whereas services represent the stuff consuming those dependencies.

    ├── packages
    │   └── cipher-test-orm
    │       └── src
    └── services
        ├── games
        │   └── src
        │       ├── functions
        │       │   ├── health
        │       │   └── hello
        │       └── libs
        └── players
            └── src
                ├── functions
                │   └── hello
                └── libs

I changed the top-level package to make it a bit easier to add new stuff without having to manually add each new directory.

    // top-level package.json

      "workspaces": [
        "services/*",
        "packages/*"
      ],

Within the new “cipher-test-orm” directory, I added a main path, which represents the entry-point for the package.

    {
      "name": "cipher-test-orm",
      "version": "0.0.1",
      "main": "lib/index.js"
    }

    "compilerOptions: {
      "composite": true,
      // ... other stuff
    }

I added the above, which I believe tells the compiler that this will be the composite of multiple different simultaneously compiled packages.

I created a top-level config file:

// top-level tsconfig.json
{
  "exclude": ["**/tests/**", "**/dist/**", "**/node_modules"],
  "include": [],
  "files": [],
  "references": [
    {
      "path": "./packages/cipher-test-orm"
    }
  ]
}

Most importantly is the references, which defines where the referenced packages live and what order they should be compiled in. This is key in situations where referenced packages might rely on one another.

Then, at the top level I ran typescript, with a slightly different flag:

    yarn tsc --build tsconfig.json

Then I added cipher-test-orm to the dependencies:

  "dependencies": {
    // ... other stuff
    "cipher-test-orm": "0.0.1"
  },

and required in the package from the health function:

import { middyfy } from "@libs/lambda";
import { formatJSONResponse } from "@libs/api-gateway";
import { repeatAString } from "cipher-test-orm";
const health = () => {
  return formatJSONResponse({
    success: true,
    string: repeatAString("test"),
  });
};

export const main = middyfy(health);

Running yarn workspace games dev locally, I was able to confirm the functioning of the component!

Woohoo.

I quickly noticed that this pattern doesn’t allow for the same sorts of easy automatic recompilation. I think it would require more work to get running out of the box.

5. Reflections

One question I have here pretty quick is whether the typescript configuration composites are necessary for all (specifically service/) files I have specifically, or if I could bump those down into the packages/ repository and have the tsc compile live specifically there.

Currently, multi-service serverless packages don’t have instrumentation to be run all at once on your local, so you can’t really turn this into a “monolith stand-in”. You have to either choose which serverless offline you want to execute, or if you want to try to spin up your own customization.

This seems reasonable to me, at least in the short term, as it’s unlikely that I’ll need to build off more than one of them at a time.

I’m getting closer to actually kicking this project off in earnest, just one more step… messing around with GraphQL and Appsync.

There’s often a debate in software engineering (well, in the startup wing of it anyway) about how important working more than other people versus being more talented or being in the right place at the right time. This has especially come to the forefront at twitter where only the "hardcore" employees are being asked to stick around. And while number of hours spent matters for a variety of professional and political reasons, I don't think it has much informative value for someone trying to be thoughtful about the work they do and how it relates to the rest of their life. How hard and how long you work often has more to do with the specific requirements and expectations of your employment than a specific decision or change you can make about your career. I believe it is more often the peculiarities and talents that have the possibility of making us successful in a given area. And, those same traits are often the ones that impose burdens on other parts of our lives.

I'm someone who ruminates frequently. Rumination, or the ability to let go of thoughts (often especially harmful ones) is part and parcel of anxiety, which I definitely have, and is also common enough among people with ADHD or other attention disorders (I have never been diagnosed, but am suspicious). It’s also the type of thing that, well, has immensely benefitted my career. I don’t know if (and I doubt) I work significantly more than other people. However, I would be willing to put a large bet that I spend way more time thinking about problems.

For example, just before this Thanksgiving break (about 30 minutes before work ended) I got a ping in a thread about a bug that might come my team’s way on Monday. I suspect for most people, this might have been a minor annoyance. For me, it meant this thing, that I thought I had fixed, was now back at the top of my head. I wouldn’t be able to settle down until I understood the why and how to fix it.

So, here I am, just before thanksgiving, when I’m supposed to be on vacation, thinking about work. I can’t help but wonder, would I be better off if I just let the problem go? It feels like a stark example of a choice between professional success and personal happiness, but I'm not sure the answer is so simple.

Problem Focus

It’s this weird overlap where my inability to disconnect from challenging thoughts all of a sudden becomes a considerable benefit. Because that thing that means I can’t disconnect from thoughts also means that I won’t be satisfied with half measures until I come out of a situation feeling very confident that I can come to a successful resolution. It means I’ll keep tossing an idea around in my brain until I’m happy with where it lands. This has helped me tackle challenges that might be over my head, helped me deliver on complex and challenging projects, and has made me a trustworthy resource across multiple teams and companies.

The Whole Quality of Life Thing

But this comes at a cost. It means that, often, I will be stuck in firefighter mode, when I should be thinking more expansively, which is a real problem. It also means I might not be spending time I need to rest and recharge, instead on continuing to plow through work. It means, occasionally, when I’m supposed to be on vacation or out of the office, I’m at my computer or at a notebook thinking about work.

Successful People Are Weird

I’d like to think, privilege set aside, I’ve achieved some small measure of success. And what I’ve learned, and had reinforced, is that successful people are fundamentally weird. One of my biggest takeaways from Cork Dork, of all places, was the idea that as you grow more focused on a certain area of your life, how you actually think and exist in the world fundamentally changes. Your brain chemistry re-orients itself around those new needs.

And while to the outside world it might look like a fully formed whole that can be broadly applied, it’s really a set of support structures combined with skill that gets them there. For example, one thing I’m doing to try to help with prioritization, is framing everything in terms of the immediate problem, so I can help myself get the urgency from the framing while still focusing on longer-term issues.

This isn’t to discount the success or say it has any less to do with hard work, just that… well… it’s really hard to draw straight line conclusions between positive and negative behavior and what will actually lead to a successful outcome. It’s easy to look at a set of personal behaviors within ourselves as things we should try to get rid of, and completely ignore what type of life we actually want to create. And this also isn't to discount the role of privilege. Often privilege is what helps create and maintain those support structures for success in other avenues.

The “Bank Shot”

The “eliminate bad behavior” (in this case working before a holiday) also creates a perverse narrative where behavioral change is as easy and controlled as simply recognizing a problem and then taking “obvious” and “concrete” steps to eliminate it.

But there was this concept that a political analyst I followed on Twitter a while back called “the bank shot”. The point was that more parts of the political process were effectively “bank shots” than we would like to admit. It’s basically this idea that things we think are carefully constructed chains of events are actually closer to someone who is trying to make a basketball shot by bouncing the ball off the backboard: something you can do repeatedly and well, but decidedly less certain and less straightforward than just taking a direct shot.

Our behaviors are like that too. Things we think we can carefully construct are often more the result of directed (but still) chance that occurs at the end of a plan, rather than a Rube Goldberg machine of outcomes. We would like to believe we can subtly sand away the less ideal parts of ourselves while keeping “everything that makes us good”. But we fail to recognize just how intertwined the stuff we get frustrated with is with the stuff we actually appreciate.

Letting Go of Problems

While I found the book How to Keep House While Drowning a bit repetitive, I did like one specific takeaway. It’s the concept that the house exists to support your needs, desires, and life, not the other way around. And I would apply this concept to things like mental health, and work/life balance as well.

Basically, where I landed is, I enjoy thinking about these problems, and I think they’re an important part of my professional goals. However, I want to be certain they don’t crowd out other critical parts of my life. So, I ended up putting together a comprehensive(ish) list of questions and potential solutions to the problem, scheduled them to send when they wouldn’t impact the rest of my team, and felt comfortable with not thinking about them, so I could fully enjoy Thanksgiving with my family.

I had an entirely different article locked and loaded about Marvel Snap, talking about the ways that randomness was being used in specific ways to construct the experience of the game. But, after talking to a friend about it, and about how we both felt the game was interesting, but a little empty, I decided I want to change tacks. The interesting part of the Marvel Snap phenomenon seemed less to be about the design choices, and more about just how many people were talking about its design choices.

If you want to read about the cleverness of Marvel Snap, you can find the commentary here, here, and here. I’m also going to be talking about games like Magic: The Gathering and Storybook Brawl, so I make some assumptions on my part about comfort with concepts like auto-battlers and trading card games.

What Is It?

Marvel Snap, released globally this October, is a head to head card game where players battle by playing cards in one of three lanes. At the end of six rounds, players sum up the power of cards they have in each lane. Whoever has more power in a lane wins the lane, and whoever wins 2 lanes wins the game. The twist of the game comes from the fact that each lane has random powers that change how the game is played, and each card players play introduces new twists, like moving between lanes, pulling other cards from their deck, and such. Players build a deck of 12 cards to take with them into battle. But the real twist of the game as it were, is the betting mechanic. Each game, at any point (until the end) players can choose to double the value of the game (the amount they will rise or fall in the ranks).

The Response

Most of the response to Marvel Snap has been broadly in one of two (very different sized) camps.

1. People who are absolutely in love with the mechanics and design choices and who feel like the game is an outstanding evolution in the card game space.
2. People (mostly indie designers) who feel like the design space is pretty restrictive and the game is ultimately boring.

What was interesting to me is how the people who loved the game talk way more about individually fascinating design decisions and less about the sum of what the game was attempting to do. And I also noticed the people critiquing the game seemed less focused on individual choices, and more on whether or not the game had enough "depth". I wanted to take some time to walk through my discomfort with both sides of the arguments. Why I think the gushing praise over the game seems more like a response to its “design-y-ness” and why also I think it’s deeper than what some of its critics might assume.

The Design Space Isn’t Too Restrictive

Broadly the argument here is that the games are so short and there is so much randomness that there can’t actually be that much depth because there isn’t enough strategy there for there to be depth. Some people also argue that the game falls into the same “Battle Pass” progression mechanic tropes that reinforce unhealthy play patterns.

The game has a metric ton of randomness. Each of the cards pushes the randomness by not allowing for much player control and certainty, and the zones double down on this by introducing new elements and avenues for battle. One zone might say “You can’t play cards here”, which could significantly advantage one player whose deck has a bunch of movement cards, or it could turn the entire fight into a two lane brawl.

I’m in the camp that this type of randomness doesn’t necessarily change how skill testing the game is, it just changes what axis skills are being tested on. I’ve put together enough highly mediocre decks and made enough incredibly mediocre plays to recognize that the ability to anticipate what an opponent is playing, and how different cards work together, is an important part of “being good”.

Furthermore, I don’t believe that there’s anything particularly deceptive about the way Marvel Snap encourages players to continue playing. All the upgrades are cosmetic, a base deck, combined with smart doubling/retreating will take you far, and the company has created a track record of being responsive to community complaints in ways that feel sincere and not exploitative. Look, I love me some Transformers Battle Arena, but it’s a free subscription game on Apple Arcade and having cards literally get stronger as you play more makes the game fundamentally un fun to play against someone who’s invested more time than you).

Higher skill players will be able to more efficiently collect cubes than lower skill players. That’s the skill of the game. It cleverly pushes the skill out of an individual match (so players of wildly variant skill levels can have fun) and into the meta structure of your ability to glean expected value from the game.

Ok, but so what?

This gets me to my next point, point one. Most of the effusive comments of the design space around this game seems to focus on how cleverly it solves a variety of problems that have plagued trading card games since time immemorial. It amps up the randomness and so keeps players engaged across a variety of skill levels, it introduces a betting mechanism to reward high skill play, it even fixes mulligans for chrissakes.

As a design object to think about and play with, I can’t argue Marvel Snap is fun to consider and toy with. It’s thoughtful, entertaining, and the amount of decision space you feel when you play the game feels inordinately high for the number of cards in your deck and your hand. From some sort of weird, scientific game design perspective, it feels like a revelation. It does that specific thing that Magic: The Gathering does so well, where the card you play actually evokes the ethos of its idea mechanically.

And I think that’s part of the reason it’s been such a gem among the designer community. It feels like a big box game that’s indie at its heart. It cares all about the sneaky mechanical things that other big box games seemingly don’t want to. But every time I get done with the game, I wonder, what did I just play? It all comes down to the damned doubling mechanic. But to explain it, we’re going to have to take a slight detour.

Let’s Talk about Magic: the Gathering and Storybook Brawl

I want to discuss these two games because they represent opposing ends of the spectrum (that Marvel Snap lands in between) both in terms of interactivity and in terms of the literal longevity of the games.

To start, we have Magic: The Gathering. It’s been around for 30 years and honestly has some of the most bullshit un-fun mechanics I’ve seen possible in a trading card game. Your ability to play cards is directly tied to lands which you have to have in your deck, making 40% of your draws on a given game basically unfun, you have to have an immense amount of knowledge experience to even start playing, and if you want to be competitive you better be ready to make a high initial investment and then also a high continuing investment as the cards rotate out of existence. The cherry on top? Those land things we talked about earlier are some of the most expensive cards because the good ones are rare. And for all of that, it’s not actually significantly more skill-testing than a coin flip. (This is a bit of an exaggeration, but I think an elite player is probably expected to win roughly 65-35). Every new and successful card game that’s come out since Magic launched 30 years ago has tried to fix one or most of its fundamental flaws.

But damn Magic is a fun game. It’s stupidly fun. Unlike other games in the online era, where laddering is a core part of the experience, just playing a single game of Magic can feel like an absolutely meaningful revelation. You are engaged in a battle of ideas from top to bottom, and your ability to survive, and win are directly proportional to your skill and understanding of your deck, the meta, and your ability to think on your feet.

Which takes me to Storybook Brawl. Right up until they were bought by FTXand I decided that it might be best to spend my time elsewhere (seriously, is Storybook Brawl still alive right now? What’s going on there?), I thought it might be one of the cleverest designs I’d seen in years, a really awesome entry into the auto-battler genre.

What Storybook does different from Magic is that it automates the actual playing of the game. Instead of playing with the deck of cards you drafted, the deck plays for you. You get matched up with 7 other players and, over the course of the drafting experience, try to eliminate other players by drafting a more coherent deck than them. Like Marvel Snap the focus has been drawn away from an individual game more towards the general meta structure of the match. You’re not trying to win a given game, you’re trying to be the best in the pool. In a lot of ways Marvel Snap and Storybook Brawl feel like spiritual evolutions of Magic. They both utilize randomness heavily (while adding mitigating factors) to decrease play time without sacrificing the interactivity of the card game concept. But for Storybook Brawl, that randomness still creates a direct relationship between your ability to play the game and your ability to get to a satisfying conclusion to an experience: your ability to draft cards and your ability to win a pool feel innately tied.

Is Marvel Snap Fun?

I will go on record and say I think that Marvel Snap is a better game concept overall than Storybook Brawl and is closer to a mobile successor to Magic: The Gathering. The aesthetic, the interaction design, and the thoughtfulness around how and when randomness gets deployed feels revelatory. But adding up all of those awesome design interventions still doesn't necessarily lead to a satisfying experience. I’m skeptical that all of the current hype has as much to do with beauty of Marvel Snap as a whole as it does with just how neat and interesting all of the ideas contained within the game are.

The example I somewhat regularly run into are games where I’ve gotten to turn 5 with some amount of back and forth. On turn 5 I make a dynamite play and go into the 6th and final round with a commanding lead, and smash the doubling cube. My opponent then makes the smart play to retreat before the end of the game, minimizing their losses and snatching some value back because they’ve seen they’ve clearly lost. This is absolutely right play, and when I'm on the other side, I absolutely do the same thing, but being on the end of this winning experience kind of really sucks.

Now the answer, for most players, will be, ”well, obviously you need to get better so that you double earlier and get more value from the game”. And from a purely “play the game better” perspective, these people are right. But my response, is “so what”? The outcome of my “playing better" isn’t me having a better experience playing that game (winning, executing better in-game strategy), it’s that I get slightly better at arbitraging my board state into a higher ranking. Why is that a compelling reason to keep at it? The betting system, as it stands right now, doesn’t help inform or improve the way I build decks or play the game, if anything, it makes individual in game decisions feel less important.

For all of its cleverness (and there is so much to appreciate) because the meta structure removes so much importance from an individual match, I think Marvel Snap needs to find another place to put complexity and meaning. Otherwise, there’s not a lot of meat on the bones there to keep me coming back.* The doubling system, as it currently stands, turns the deck on deck combat into an efficiency engine to grind for cubes and rewards, and not much a marker of much else. And that’s really disappointing.

It feels possible that there’s a great competitive play experience waiting in the wings to really unlock the experience of the game. But right now, it feels a bit more like a small, sweet abstraction of heavier and more meaningful games that will continue to dominate the genre.

* Typically when I play Marvel Snap it reminds me of how much I want to play Magic: The Gathering, but then I remember how expensive that is, so I play a single game of Snap and go do something else instead.

I have a lot of what I’m going to call (tongue-in-cheek), toxic, leadership beliefs. People who have status, I believe, should be held to a higher standard of behavior than people with less status. I decided to collect some of my thoughts here:

My Four Most "Toxic" Leadership Beliefs

• The Leadership Management Dichotomy is BS
• Your Status is your Stake not your Investments
• Leadership Isn’t For “Best Intent”
• Servant Leadership is not "the only way"

The Leadership Management Dichotomy is Bullshit

There's been a huge effort in the last ten years or so to draw a huge distinction between being a leader and a manager. A lot of effort has gone into this idea that leadership is an ideal to strive for while management is... the end result of bureaucracy. It is true that people can lead without being a manager. That’s, in fact, the entire point about positions like “Staff”, “Principal”, “Architect” roles, and positions like “Product Management” in the tech field. You don’t have a direct say in what people are or aren’t able to do, but your goal is to get them to do certain things, nonetheless.

But, contrary to popular arguments, you cannot be a manger without being a leader. That sort of dichotomy that’s been sold by “thought leaders” who would like to throw away classes of management because you’re “just managing” and not truly “leading” is creating this wave of behaviors that one attributes to the “bad way” management, versus the “good way” leadership.

The example is always something like “leaders lead through inspiration” while “managers lead only through authority”. Like, the absolute worst. Sometimes you need people to do things to keep the lights on, and the goal is not in fact there to be inspiring. It’s because someone has to do cleanup. That’s management and it’s leadership.

The dichotomy is there to paint the perspective of a world where change only happens through big moments and ideas and not through hard work and slow progress and iterative improvements. It’s such an impoverished way to look at the world. We can have an honest conversation about what makes someone a “good leader” and talk about whether or not being inspiring is a core component of leadership, but to separate certain behaviors out into being “merely” a manager is to short change an entire conversation with a false dichotomy.

Your Status is Your Stake

The way I see most people treat professional growth is an investment that accrues over time. (Like a house, a stock portfolio, or a savings account) This leads to a lot of behavior that’s default conservative, protecting the status at the cost of growth, learning, and team development. This sort of perspective treats leadership as a general, universal role we ascend into, instead of a specific moment in time that we’re at.

I say this because we often treat the “do nothing” position, especially as we accrue more status, as the safer option. The “don’t speak up”, “don’t rock the boat” option as more protective of our investment. But… from a betting perspective, “do nothing” is just another form of bet, not different from any other risky action.

I think we shy away from this because that perspective makes it all seems so much more… tenuous. But that’s what it is. Your value as a leader is dependent on the social, political, and financial structure of the company you’re at. And maybe, if you want to continue on that path, there will be other similarly structured companies that will be interested in those skills. But status is a bet. For example, your ability to be an effective software engineering leader probably has minimal (or less) overlap with your ability to control a classroom of elementary school students. Because of the relative values our society places on those roles, one is more important (or more financially viable) than the other. But if that importance was suddenly flipped, I would not expect the software engineer to necessarily jump into the teaching role, purely because of previous status.

Often the people who most recognize this sort of picture aren't the people immediately next to you on the org chart, they're the people who have less status. As you get more defensive and more conservative about your viewpoints, a wider gap appears between the world as you see and the world as it exists. This is because your status insulates you from those challenges, and if you never stake it, never put it up for grabs, you're never actually exposing yourself to the sorts of risks other people experience more frequently. This is especially possible at larger orgs because the risk profile on most decisions is so low, and the cost of that emotional gap for the people with less status than you is close to zero.

But, I believe each decision you make as a leader is a bet. It’s chips on the table arguing that a particular vision for the future will come to pass. If it doesn’t, those chips go away. That can be socially, in the form of reduced trust, or literally, in the form of being put on different pathways for your career at a company. Status as a “stake” also comes in the form of your ability to martial forces socially. If you are preaching a vision for your team or your division or your company that never comes to pass, it doesn’t matter how much your formal authority remains, you are losing that bet with the people who follow you as they see the difference between the vision and reality.

Best Intent Shouldn’t Apply to Leaders

There’s the saying, especially at tech companies that you should “assume best intent”. The gist goes that in situations that are unclear or where you believe someone else has done something that makes your life more challenging, it’s likely the result of a misinterpretation or misunderstanding between you and them.

During an earlier period of my career, I walked through a conversation with a boss where they argued that I should look at what I considered “unideal” behavior or outcomes on the part of senior leadership through the lens of “assume best intent”. Basically, I should assume that they weren’t trying to take action to make my life worse but trying to take action that they thought was optimal.

My boss wasn’t wrong to share this perspective, but I pushed back, and I continue to believe that it’s not a particularly useful ask or expectation when coming from leaders to the people they are leading.

Beyond the general problems with “assumption of best intent”, there are specific problems with leaders asking for an assumption of best intent, I want to specifically deal with the situations where leaders are often in new (and very uncertain) scenarios, asking for “assume best intent” from the people they lead.

I’m going to come at this from the perspective that this is a mostly functional organization, and not one where leaders are actively trying to take advantage of the people they’re leading. I think the request for “best intent” comes from a very sincere (and sometimes vulnerable) place. Leaders recognize their flaws and are asking for patience and understanding from the people they are responsible for.

I think it also comes from a place of reassurance. They want people to know that while things might be bad now, they're bad because of thoughtfulness which will hopefully change in the future. This is another area where "show not tell" is important. Company priorities change all the time, which can have detrimental effects on people in other departments. Starting with "we're being thoughtful about this" doesn't actually suggest you're thinking about and responding to the needs of the people who you're talking through it with.

Well, in my opinion best intent short circuits necessary conversations, responsibilities, and ownerships that leaders need to take to either regain or retain trust of the people they lead. When you mess up, what defines your ability to maintain the trust of the people who work with you isn’t whether or not you went in with good intentions. It’s driven by your ability to show thoughtful understanding of what happened, how the people around you felt about it, and what action you plan to take in the future to prevent it from happening again. And then, taking the necessary steps to show that growth as time goes on.

The world where you’re taking those actions, best intent and worst intent are somewhat irrelevant. My goal here isn’t to argue that you should assume worst intent, far from it. It’s more to say that when leaders are asking for forgiveness, understanding, or a retry from negative outcomes, they have to recognize that the framing around their process and their results is entirely different than it might be for someone else.

Rather, I'm saying that, if you want to be a good leader, best intent is where you should start, not a valuable question to introspect how well you're accomplishing your job.

When you’re an individual contributor, the outcomes your produce are contingent on being pointed towards the right goals. If you effectively execute “the wrong thing”, you still have value as a contributor. (Look I think you should care about doing the right thing, but it’s not the standard you’re being held to) The further you go in leadership, the less those sorts of “we tried our best” matters.

Servant Leadership is Fake

I should preface this by saying I believe in the concepts and the tenets of servant leadership. I think the writing of people like Adam Grant are valuable and worthwhile and worth following. But, the concept that servant leadership is the #onetrueway to lead a business is as much a reflection of the location and the society that you operate within as it is about the company you happen to be at.

From a business perspective, your goal as a leader is to use your brain to significantly move the lever on profits for the company. If that lever comes in conflict with other people’s jobs, behaviors, happiness, your job as a “leader” is in fact very clear. High trust societies And I don’t think we talk about that honestly enough when we talk about what that means when those two goals come into conflict. To bias one or the other says as much about what you believe your community's goals should be as it does about what is the "optimal" outcome for the business.

I think when many of us recoil at what we’re seeing at twitter, we’re viewing things from this very long term lens of “this can’t work because it goes against what I’ve been taught about how to treat teams right”. And… that’s not wrong, but it’s also looking at it from the lens of like “how can we make this a successful, socially relevant company 5 years from now”. There are other (capitalistically) valid ways to successfully run a business. They just happen to suck for a vast majority of the people involved. Even from a purely profit driven perspective, the types of decisions you make often rest on whether or not you're optimizing for a short term or long term gain. And, servant leadership often looks the best over the longest time horizons.

Our decision to choice into specific types of leadership roles that align with our beliefs, is a decision we make about the quality of certain types of leadership and the world we want to build. Being a leader that values our communities isn’t just a statement about “what’s most effective”, it’s a statement about the type of world we’d like to build and what we’re willing to sacrifice to see that through. When we talk about “servant leadership” being “the one true way” we’re short changing and devaluing the fact that it might be the best way within the society we want to live.

I was promoted into the Staff Engineering role last October (2022). Coming into it I've read a small portion of the few books that exist related to staff engineering, specifically Staff Engineering by Will Larson and The Staff Engineer's Path by Tanya Reilly. I've also been really enjoying a couple of articles by Joy Ebertz on the topic.

My Initial Thoughts

One of the reasons I wanted to write this is to document my early thoughts before too many of my opinions become ossified by the interactions I have with other people.

• Everyone has a different vision for staff engineering
• Code and Culture
• Getting Here and Going There
• My Theory about Wealthy People
• First "Bullshit" Role

The DALL-E Prompt for this was: someone who is a very important software developer but tries to stay humble

Everyone has a different vision for staff engineering

I've asked 3 different managers, multiple other staff engineers, and architects how they've been thinking about my role on the team, especially as it pertains to how much code I should be shipping and in what way I should be most effective for my team. I've gotten a unique answer for every one of them.

Some people say Staff Engineers should ship roughly the same amount of code as senior engineers but within a much more efficient time frame. Others say staff engineers should provide a bigger focus on setting other team members up for success and less time on code. And others talk about the divide between making an impact on the team and making a cross-team or cross-org impact.

Based on the small amount of reading I've done... they're all basically correct.

Which leads me to my major theory...

Code and Culture

Staff Engineers are responsible for shipping code that has outside impacts on company culture.

One of the main focuses of reading I've done in the last 5 years has been around how much CI, deployments, and testing processes have an impact on the team. Often your team culture will fall into the limits and supports described by that system. Basically, as a senior engineer, any time you're making the same sorts of arguments over and over in a pull request, you should find a way to automate that and put constraints that help the team perform better.

Staff Engineering is like that, but with a much sharper look at specific cultural structures that can be unlocked by the sharp application of tools and code to the problem. It's not just about seeing code issues or cultural issues, it's about recognizing the highest leverage issues you can and then hammering on them until something beneficial comes out.

My experience so far is often this means putting myself into situations where I know the least, so I can gather the knowledge and provide a semi-better trodden path for the rest of the team.

One thing I've noticed is a tendency to fall back on processes and agreements. And, as always the political stuff is a part of the system. That whole bit about building narratives and getting buy in is super important. But I truly believe at th end of it Staff Engineering is about producing something tangible (even if it's quite small) as a pivot point for other developers to operate on and against.

Which leads me to...

Getting Here and Going There

One thing that's stood out most clearly as a role that I'll need to give up is how much urgency I act with.

In my role as a Senior Engineer, the fact that I treated issues with importance, urgency, and care was a primary reason (I believe) I was able to make myself a valued member on the team quickly. But, moving to Staff Engineering I'm starting to see how the stakes of that game have changed rather dramatically.

Firstly, it's the realization that if I'm acting with incredible urgency on most tasks, I'm not giving myself the time to think about those high value things I talked about earlier. Secondly, if I'm taking those tasks I'm taking up valuable opportunities to lift the rest of my team to learn and grow. And finally, acting with urgency conveys a certain level of importance given the title and my team will likely map to my behavior. By giving things space to breath I'm actually sending a message that we can plan, think, and react to events as they come in, rather than rushing to solve every minor problem as it shows up.

Which leads to a weird thing...

My Theory about Wealthy People...

And how it applies to Staff Engineering. One random thing I've noticed from the Elon Twitter saga, and also watching way too much Million Dollar Listing, is how much the problems of wealth change and shape the way you see the world. Specifically, as more and more of your problems can be solved by throwing money at them, the problems that people seem to encounter are the problems that are constraints more closely aligned with our current scientific, political, and social reality. I.e. problems you can't just throw money at.

Similarly, moving up the "ladder" at work into Staff Engineering I've noticed similar constraints. There are very few actual constraints on what I have permission to do. If I just go off and work on a project for 2 days, I will get a wide swath of trust that what I'm doing is valid, correct, and adds value to the team (even if I'm just derping around). But conversely, "just doing stuff" isn't actually going to move the needle for me emotionally or practically. Most of the constraints I've noticed are the hard political constraints of building momentum for a course of action, getting team members bought into a new way of doing things, or trying to implement a new technique/technology within the constraints of the old system. There just aren't that many if this then thats left.

First "Bullshit" Role

The final thing I've noticed is that Staff Engineering is the first role where coasting feels... possible. This isn't a subtle critique of current or past employers, it's more a recognition of the nebulousness of the role and the ease with which it would be to fall into its traps. As a Staff Engineer I'm no longer as directly tied to my ability to commit code. If I don't code for 3 months it's easily conceivable I'm working on a big picture sort of project that's high leverage but long term. And I'm also not responsible for people the same way as a manager might be. If my team is hypothetically struggling, I can give feedback to my manager, but ultimately my role is to help raise the level of my team not necessarily hold them accountable.

It's a weird, scary, and unconstrained place to be, and something I'm going to have to learn to live with. If leadership is about :staking your beliefs, then dealing with the uncertainty (and the potential for bullshit) is part of the stake. Learning to make my own hypotheses, argue for them, and try to align myself best to the business and my teammates is the thing I have to do.

Let's See How This Goes

I'm excited to look back on this in a couple of years and see what sorts of things I got right and wrong. Here's to an exciting 2023!

There’s a lot of discussion about how as an individual contributor it’s important to “make your work visible” to higher ups to get the appropriate resources, opportunities for advancement, etc. But there is relatively little discussion going the other direction about being a leader and making your work visible to the people whom you lead. I think there's just as much value to thinking about and framing your work for the people lower in level as there is to higher ups.

In my last post, I aggressively (and somewhat sarcastically) referenced Staff+ engineering being “:bullshit”. But, I don't want to imply that leadership is bs. Rather, I want to point out that as a leader, especially from the view of the people you lead, you get to both make up the rules and then evaluate yourself based on those rules you made up. From an external perspective it can make it all seem a bit... arbitrary, and designed to post-facto argue why you are doing a good job, regardless of the "truth on the ground".

And it's true, in other roles it was easier for my collaborators to evaluate the job I'm doing. But in the Staff+, and other leadership, roles that moment-to-moment determination is solely up to me, depending on how I feel about the situation and what, I think, would be best. And the line between doing something that’s incredibly self-serving, and something that’s beneficial for your team can be nebulous and is often the result of a personal judgment call.

This is the power and risk of the role. As always, the easiest person to fool is yourself. But we often talk about the risk of the role from an upward looking perspective, i.e. aligning yourself appropriately with the goals of business leaders and managers. But what about looking the opposite direction? How, given all of that uncertainty, do you maintain the trust of the people who follow you?

The DALL-E prompt for this was a cartoon hippo in the middle of tall grass with onlookers from a 3/4 bird's eye view.

What is Being Visible?

There’s some popularity in tech writing that talks about the book Seeing Like a State. The general concept here is that control structures will ask for behaviors that are observable and understandable. In doing so, reducing positive development by pruning things they don’t understand. This broadly applies to tech orgs with things like OKRs and metrics.

When I talk about “making yourself visible” as an IC, it means framing your behavior in terms of phrases and logic that fits with the goals and priorities of your management.

When we mention upward visibility, the career benefits are obvious, better opportunities for advancement and more interesting problems to solve. The value of downward visibility isn’t clear because the consequences aren’t obvious. If the people who you lead don’t know what you’re doing for them, there is no career cost to you. There might be an efficiency cost to the team, but it’s unlikely to immediately have an impact one way or another.

I’ve had a decent number of conversations with coworkers from past tech jobs who ask questions like “What does X director even do”. These discussions have made me notice how as we move up the leadership chain, we lose sight of making our behavior visible to a different set of concerns, the other ICs who live within the system.

A Hypothetical Situation

Let’s say I, as a staff engineer, work on a team struggling to deliver a particular project. I have a few options. I can:

1. Note that there’s an issue, but stay out of resolution determination with the team
2. Communicate resolution steps to the team, but not take part in resolving the failures
3. Communicate with the team and take an active role in making the project a success

In various points each of these outcomes could be the exactly right thing to do for the team, it’s also possible that #1 could be a complete copout that sells the team short for my failings. It’s also possible that #3 could mean I end up overshadowing the team. Maybe we feel great that we launched, but there’s resentment that now people outside the team see the project as mine instead of the team’s or the other project leads.

From my perspective, good leadership requires a constant sort of vigilance in terms of reflection and understanding of my behavior. Even asking for feedback from the people impacted won’t necessarily be a good barometer. People could be happy I jumped in and fail to see the negative externality I just caused on their career. Or, people could be unhappy I didn’t jump in, but fail to see the way the career growth and positive feedback they got for their response to the challenge improved their careers.

Note: This isn’t to say that you shouldn’t ask for feedback. It’s to say that the feedback is non-determinative in a way that it’s not in other roles.

This is the exciting and terrifying thing about staff+ roles. You are fully in charge, and can exceed or fall back to the level you feel comfortable. The constraints can be less about practical outcomes so much as they are about your ability to tell a good story and your desire to see impact.

It’s hard to manage this visibility, and this challenge as a thoughtful tech leader (which I aspire to be, though I suspect I fail at quite regularly). We value the contributions of our team members, we view them as our equals, and we don’t want to put ourselves above them. The difference between being open and transparent and flaying your emotions out at the people whose careers you impact can feel awfully thin.

Being Visible to the People You Lead

It’s easy to create this picture where everyone is focused on the same set of concerns and metrics that are set and driven by leadership. And it’s true, we all want to do well for the company, but the things we all value have niches and twists. Some people might care most about doing well for their users via accessibility, some might be driven by learning the latest technology, and others might be driven by things like optimization. Each of us is going to interpret company goals, and the best way to achieve them, in unique ways.

It would follow, then, that making yourself visible “downwards” would firstly be about framing your impact in a way that’s most interesting and valuable for those goals, not just the company ones. Most importantly, especially in an IC leader role, you can attempt to reframe the concerns of the company in terms of the concerns that your team has. But doing so requires thoughtfulness not just about how to hit goals, but also listening to what the team values and how they can achieve those values within the framework the company provides.

Being “Visible” is Easy as an IC

As an IC slash leader, it’s easy to default back to shipping code, so you feel valued, and so that you can “show value” to the rest of your team. You can always pick up tickets! But as Will Larson makes clear, that’s much more akin to snacking than providing actual value. Actual value might look more like doing the hard work of explaining to your team (and encouraging them to challenge you) on the 3+ month vision doc you’re building out. It’s certainly possible that working that ticket is the most important thing you could be working on, but even so, I think you should have an answer beyond “I’m a developer, and I’m shipping code”.

What Does Being Visible Look Like

Let’s go back to those three outcomes for a flailing project. How can we make this visible:

Firstly, you should be transparent with both your manager and the people involved in the decision what tact you’re taking. There isn’t a situation where you should feel like if you clearly communicate what’s happening, your team members won’t “get the learning” so some silliness. You should also talk to the team and understand how they’re feeling about the project, what concerns they have.

In each situation, it’s worth concretely calling out the following:

1. The concerns of the broader organization
2. The goals of the team
3. How you’ve come to the conclusion about integrating the two, and where you’ve picked one over the other, and why

**If you don’t think you should be involved in problem-solving the issue: ** You should explain that you have concerns as an external participant, but it’s not your place to get involved with how to solve it. Also make yourself available for feedback, but refrain from being an active participant in the discussion. Be clear that you either don’t have the bandwidth or think it’s valuable to not be involved in the solution because the right people are already involved, and you have high confidence in what they’re doing.

If you don’t think you should be involved in the execution, but you have real thoughts about what the key next steps are I would start by listening to hear what the rest of the team has to say, and then finding ways to piggyback and reinforce their ideas. If there are any big misses jump in and say, “hey have you thought about X, Y, Z, I have some concerns”. But then be clear that you either don’t have the bandwidth or think it’s valuable to not be involved in the solution because the right people are already involved, and you have high confidence in what they’re doing.

If you think it’s important, think you need to get involved, and are going to do so, I think this is a place where signaling intent is crucial. You likely have a lot of knowledge about the rest of the org that your team doesn’t. Be clear about why you’re getting involved, acknowledge the potential costs but signal why you think the tradeoff is worthwhile based on your knowledge of the organization’s vision, and your team’s values. And listen to any feedback your teammates might have about why they might disagree.

Rebuttals

Outcome focus and our discomfort with talking about ourselves

One counterpoint to all of this is the concept that, no, in fact, as a leader, your focus should be on outcomes. If the team has good outcomes, it doesn’t matter if they see your hand in them all the better, it means you’re making a quiet impact. And furthermore, to call out all the great stuff you’ve done is just self-serving, you’re basically asking other people to appreciate you for the work you’re responsible for!!

Here’s my response:

No one is ever 100% in agreement about what outcomes are good/bad and in between. Making them present makes it more possible for your team to nudge or redirect you in positive ways.

Invisible improvements don’t help create a good sense of team culture or reinforce the fact that you’re making efforts to improve the team, as you see it. Sharing what types of outcomes you’re concerned about makes you less of a “shit catcher” that blocks your team from the rest of the org and less of a values oasis: and more of someone who helps your team members have access to the rest of the org as they see fit. Maybe they’ll see something in an outcome you’re focused on and offer to step in and help out.

Sharing your map and your work also helps other people orient themselves more in beneficial ways for their careers. If they know what you care about, they can frame their contributions in the same way.

In the above example, maybe let’s take the “hero leader” path: You do what’s best for the team’s growth and tell them you trust them, it’ll take how long it takes, and you support them 100%. However, you don’t tell them there’s broader concern among the org and this is actually a critical release. In that case, you’re denying them crucial information about the constraints of the project, and the costs you are exposing yourself (and potentially them) to as a result.

Or maybe, let’s go the opposite direction: let’s say it’s your view as a leader to always fall in line with the company position, even if the team disagrees with it. Not acknowledging the conflict doesn’t make it go away, and might make the team lose trust in the decision process. Obviously, as a leader, you have to ultimately see that the organization’s goals are carried out. But, you have an opportunity to honestly talk about how you recognize the potential conflict in values, and how you can see a productive path forward.

But the spotlight already is on me

The other rebuttal to this sort of “leaders aren’t visible” argument is the concept that, as a leader, everything you say and do is already under a microscope. People look at you for stuff like behaviors, what to do, and what the priorities for the team are.

For me, that’s more “leader as a pane of glass” than a leader whose work is visible. The difference is between saying something like “We’re focusing on X this quarter” and “Here are the key viewpoints, insights, and feedback that led me to have the team focus on X this quarter”. If we flip this on its head, it would sound equally ridiculous as a tech leader to tell them that you’re going to only focus on tech debt because “it’s important”. The goal of making things visible is framing, reframing, and integrating one set of value systems to another, and listening to feedback to improve and manage that process.

Hippos in the Grass

In high school, I remember learning about how in the wild, the grass is tall enough for hippos to hide in it. The lesson was that just because you can’t see the grass move doesn’t mean that a hippo isn’t there ready to charge. I think, especially for thoughtful tech leaders, there’s some reticence around recognizing our own hippo-ness. We value the contributions of our team members, we view them as our equals, and we don’t want to put ourselves above them. But often we do this by talking less, by decentering ourselves too much. This can make us like the hippos in the grass, where we are still there, still dangerous, but hidden. It is safer for us to have our behaviors out in the open, visible to the people around us. This way our teams can see and respond to them, with the availability (and encouragement) to critique and shift our thinking, letting them take advantage of the influence we have within our orgs for their and our betterment.

So much of how we think about technology is predicated on the concept of a conversation, you do a thing, which creates a reaction that ends up in a result. But distributed systems don’t actually behave that way, they’re an entirely separate beast that doesn’t directly conform to traditional ways of thinking about their operation.

I’ve hit upon a metaphor that has held up and helped me that I think it’s worthwhile to share. Distributed systems (and my job) is like designing and maintaining a baggage system at an airport.

Why I don’t like the restaurant metaphor

The most common metaphor I’ve seen for distributed systems is a restaurant. And it’s the most obvious for a variety of reasons. It’s something we interact with a lot, it uses things like queues and orders, and issues of timing come up all the time.

But… I don’t like it because it doesn’t cover the thing I actually care about, which is the ability to describe how things fail. When things go wrong at a kitchen, as far as I can tell, the metaphor of a distributed system stops working quite so well. This is because kitchens still operate more akin to a series of tightly coupled components, than a distributed system with multiple concerns.

I also appreciate the Airport/Baggage metaphor here because, as someone who has checked baggage, I know there’s an input and an output, but I can’t actually visualize what’s happening.

There are a few reasons I think this is a helpful metaphor:

1. It’s been visualized by greats such as Toy Story
2. It’s a complex system that we know is complex.
3. It scales really well
4. It helps explain the concept of cascading failure
5. Visualizing Cascading Failure

DALL·E a bird's eye view of the inside of the denver airport's baggage system

1. It’s Been Visualized by Greats Like Toy Story

Firstly, and I cannot emphasize this strongly enough, I don’t have any inside knowledge of how a baggage system works.

https://youtu.be/JxHXsjMIvD0

That’s part why I think it’s helpful. The goal here isn’t necessarily to create a perfectly accurate metaphor, it’s to use this metaphor as a tool to help people (me and others) visualize the benefits and components of the system.

When you think about a baggage claim system, there are a few components that jump to mind:

1. Baggage Check
2. Bag Tagging
3. The Carts that Deliver bags to loading
4. The ramps that baggage people throw bags onto to get onto the plane

We can also recognize this as a vast and complex system with many competing demands.

2. It Scales Well

This actually might be my favorite part. One of the most difficult challenges, especially among engineers, is understanding and conceptualizing the jump from a “Server” to a “Distributed System of Microservices”.

The server pattern is easy to conceptualize. It’s got endpoints, you make requests, it does some stuff with a database, and then you get a response. To me, this is very similar to the goal of a small regional airport. For the most part, there aren’t many moving pieces. There are only 1–2 flights going in or out at a given time, so what you’re trying to accomplish is more straightforward. You store the bag in a holding location, someone comes and picks it up, and then they deliver it to the plane. And the same process happens in reverse when they unload the plane. It’s direct and it’s one to one.

It’s also apparent this is not how a system like the biggest airports in the world function.

3. It’s a Complex System that We Know Why It is Complex

Thinking about the baggage claim system of something like O’Hare and the types of problems it has to deal with makes it clear why the “direct” model wouldn’t work all that well. To process that many bags that quickly, it simply can’t rely on doing everything inside a single process. It needs to distribute the load between different components, each with different concerns.

It’s not too hard to imagine that part in the Toy Story 2 scene of the bag switcher as a small microservice. It doesn’t know anything about how the larger system operates, it has no idea where the bag might be headed now or later. All it cares about is making sure it kicks the bag to the right place.

Visualizing a software system in such a way can help make it clear why they’re so complex. Often when we interact with a website it feels immediate, like a call and response conversation. That makes it seem like what’s happening is simple, but usually there’s a lot of abstraction to go on under the hood to make it seem that way. The comparison to an airport is helpful here because the outcome is also simply (make a bag go from point A to point B) but the complexity of that task, with all of its competing demands, is clear.

4. It Helps Explain the What of When Things Go Wrong

The other reason I prefer the baggage metaphor to the restaurant metaphor is that it helps better explain what happens when the system breaks. Cascading failure doesn’t… really… happen at a restaurant, at least not the same way. Maybe our order gets delayed or discarded, but those are all easy to see and conceptualize, and not really the types of bugs that happen in software.

Often bugs aren’t even seen by the people they may have impacted, or we hear them as a weird one off story. We all have heard some story of how someone’s bag got on the wrong flight or their flight was delayed, but their bag got there on time. These are stories that map easily onto the types of issue I deal with in my job. I’ve had bugs where some data that needed to make it from a service I own to a service owned by another team, and it failed to get there for some reason.

But for the most part, when the system works, poor operation isn’t ever seen by the user, it’s just experienced as a bad day by the people who operate the system. And this is conjecture, but I suspect something similar happens at airports all the time. For example, when we’re frustrated at an airport baggage check-in because there’s a delay, there could be many systemic reasons going on behind the scenes.

5. Visualizing Cascading Failure

Looking at you Southwest…

DALL·E Edward Hopper painting of hundreds of bags at baggage claim and disgruntled passengers waiting

These systems are so intricately linked and well maintained, often the only times we see things go wrong is when they go… very wrong. That’s really the only time we get an inner working into all the things that’s happening behind the scenes.

The JIRA issue of last year(https://www.atlassian.com/engineering/post-incident-review-april-2022-outage) and Southwest’s issue over the holiday(https://www.cbsnews.com/news/southwest-airlines-missing-bags-passengers/) seem pretty similar in this regard.

And this is the final component of the metaphor, the types of concerns and issues you face working in that complex system, look very different from the direct response world. It’s a personal reminder when building out that a distributed system has a variety of wholly different concerns than building out a more direct application.

What’s a Meta For?

I think there are three primary takeaways here:

Finding the metaphor has been personally quite helpful for me to explain what I do and what problems I solve to my friends and family.

But mostly, it helps me reframe my perspective about what I’m trying to accomplish with the software systems I design. There will be times when I get frustrated by the amount and pace of the concepts, terms, and different ways of viewing my job. It helps to ground myself by thinking about whether the software I’m thinking of fits at an “O’Hare” or a “regional airport”.

Finally, it serves as a reminder for the fact that, similar to how there are plenty of regional airports, there are plenty of use cases where building out a more direct server pattern makes a ton of sense.

AWS doesn't really let (or arguably want) you to develop functionality locally. There are probably some justified, and selfish reasons for this.

The justified reason is that in a storage first model for software development, running something "locally" means spinning up the entire set of services and containers so that you have a functional system. This is what localstack aims to do, but it's understandable why this could seem to be a bit much. For example, if you're building a frontend that consumes data that's the output of a step function triggered by an event, ostensibly all you need to do is to connect to the data layer, but in order to get the data in you either have to a) set everything up or b) spoof the data anyway, so why not just mock it in the first place?

The less justified reason is that AWS makes money off of all those billable minute, their goal to serve you products and their desire to have you "build in the cloud" is well-aligned. If they make it really easy for you to "build in the cloud" they get those sweet sweet developer when you're figuring the code out in addition to when you're deployed to prod.

My understanding of the purpose of AWS SAM Sync is that it aims to do some of the stuff of making direct development possible. But instead of creating a virtual machine on your local

It's a Cloud Formation... Get it? Photo by Resource Database on Unsplash

The Links to the Actual Docs

Figured before we went further I should just link directly to the SAM Sync docs:

The Actual SAM Sync Docs

Big Picture Goal

My big picture goal is to test out whether or not sam sync is a viable solution for testing and checking development if I wanted to build out a pattern that's a direct connection from Appsync->DynamoDB.

The theory is that encapsulating more logic in a appsync resolver, instead of a lambda will remove cold starts entirely and allow you to easily encapsulate logic.

Could I say... use sam sync to quickly build out a new set of resolvers without much lift? Or... spin up a quick interface if I wanted to test out an interactive front end against it?

Today's Goal

But for now, the first goal is... can I get the :cipher-sam repo I had set-up working with SAM locally. Nothing fancy, no changes to how it currently works, just using SAM.

My Steps

Well first I did the most obvious thing possible. I ran sam sync in the repo:

$ sam sync
Usage: sam sync [OPTIONS]
Try 'sam sync -h' for help.

Error: Missing option '--stack-name'.

The issue here, thanks to the error output is rather straightforward. Name it! My suspicion is that if I were working on an actual project, I would probably want to name this something specific like service-pr-# or service-my-name-pr-# or service-dev-my-name-pr-#. But... I'm in a personal AWS account so I yolo'ed it to the name of the project cipher

So I ran

$ sam sync --stack-name cipher

There was a bunch of CloudFormation stuff, but ultimately got yelled at for the following (obfuscated) error

aws:iam::########:user/Name is not authorized to perform: iam:AttachRolePolicy on resource: role cipher-HelloWorldFunctionRole-######## because no identity-based policy allows the iam:AttachRolePolicy

This is one of those times where I could tell you that I easily figured out the results because it's right there. But I spent a good 30 minutes trying to remember the exact configuration of how AWS IAM works.

It was basically:

1. Create a new Policy
2. Find the associated Service, Action, and Resources (I got tripped on this a few time because I thought the service would be lambda, but the Service in this case is iam, for what I was trying to do, and the Resource were specific ARNs that would limit the scope of the policy)
3. Attach that policy to the appropriate group/role that was running my sync action locally

Ok cool... so deleted the stack from CloudFormation and reran

$ sam sync --stack-name cipher
...
...
Stack creation succeeded. Sync infra completed.
$

Sweet. I went and hit the endpoint provided and I got back...

{"message":"hello world"}

Awesome... lemme go make a quick change to the response and retry... and...

still

{"message":"hello world"}

I'm being a smidge facetious here because the output above from sam sync ended, which kind of suggested there wasn't an active process. But yeah... this is where the AWS docs came in handy. I did a quick search for watch and I found this section:

--watch Starts a process that watches your local application for changes and automatically syncs them to the AWS Cloud. By default, when you specify this option, AWS SAM syncs all resources in your application as you update them. With this option, AWS SAM performs an initial AWS CloudFormation deployment. Then, AWS SAM uses AWS service APIs to update code resources. AWS SAM uses AWS CloudFormation to update infrastructure resources when you update your AWS SAM template.

Perfect!

Re ran with

$ sam sync --stack-name cipher --watch
...
Stack update succeeded. Sync infra completed.

CodeTrigger not created as CodeUri or DefinitionUri is missing for ServerlessRestApi.
Infra sync completed.

I'm going to ignore that warning for a second, and I just went ahead and retried making a change to the response body. This time saving resulted in the following in my terminal:

Manifest is not changed for (HelloWorldFunction), running incremental build
Building codeuri: /cipher-sam/hello-world runtime: nodejs16.x metadata: {'BuildMethod': 'esbuild', 'BuildProperties': {'Minify': True, 'Target': 'es2020', 'EntryPoints': ['app.ts']}} architecture: x86_64 functions: HelloWorldFunction
Running NodejsNpmEsbuildBuilder:CopySource
Running NodejsNpmEsbuildBuilder:LinkSource
Running NodejsNpmEsbuildBuilder:EsbuildBundle
Finished syncing Lambda Function HelloWorldFunction.

Sweet! I went and hit the endpoint and got...

{"message":"hello world bob"}

Exactly what I wanted! And pretty fast too.

But before we left... I wanted to try one more thing, what would happen if instead of changing the code, I changed the route? I went ahead and changed my /hello endpoint to /bingbong

This time as I guessed, the change was much... slower. Because I was changing the actual AWS infrastructure it meant going through a bunch of CloudFormation functions like creating a new resource, creating the permissions, and then deleting the previous resource.

One more catch

The other thing I'd like to add is that sam sync --watch won't tear down the environment for you when you're done with it, you have to either run sam delete or go into CloudFormation to delete the stack. (You will also make sure your user has the right access points to both create the roles/policies and also delete those roles/policies)

What's Next

Ok so realistically this was a pretty basic test right? But I was kinda curious what the extent that watch would quickly deploy changes and honestly (again this being a basic example) it was pretty good! If my expectation was I had a pretty simply Backend For Frontend that integrated to a broader system, it wouldn't be all that hard to spin up for a quick test. With the caveat that I would probably (as a hypothesis) mostly use it for making straightforward code changes, more direct debugging, or just needing an interactive interface, I would not use it if I were trying to test out options around APIs or other infrastructural changes.

So what's next?

My next goal in this journey is to replace my Api Gateway Interface with an Appsync interface. I'm curious if that will make any changes to how fast this thing spins up!

In mid through late 2021 I experienced what I would call burnout, the concept that I truly had nothing left in the tank emotionally to strive at my job, even if the interest remained. I still remember the vague googling around “recovering from burn out” in late 2021. It was… so disheartening. Looking back as someone who's found quite a bit of happiness and contentment with their job, I decided to write the article that might've helped me feel less alone and confused at that time.

Most of the advice revolved around “rest and recovery” and “taking a break”. I think it all stems from the popular conception of burn out, that you burn out if you work too hard for too long.

But I’d be worked 80+ hour weeks to get a launch out, and then feel “burn out” only to take a small break, get some good sleep again and feel ready to go. No, this was something else. Those sessions felt more like doing a really challenging workout and needing recovery than burnout, which felt more like an injury.

And around the same time, Adam Grant popularized the term “languishing” for the sense of aimlessness that knowledge workers felt during the pandemic. The loss of direct community and direction. But this wasn’t that, I had re-found a community of purpose at Guild that I was interested in, and I had specific goals I would like to achieve.

Finally, I knew what I was going through wasn’t depression because the desire was there. It wasn’t that I didn’t want to push hard or do good work, it was that when I started pushing, waves of emotional exhaustion would hit.

Having been through severe anxiety before, it was possible to differentiate between what felt like a serious mental health condition, and honestly, in retrospect what felt like an emotional work injury.

A Dall-E Representation of: the internal experience of burnout represented as a cave painting

How it Started

In late 2018 I left my first job as a software engineer to join a small startup named Pana that was trying something compelling in the business travel industry. They were a company focused on “Guest Travel”, or travel that was still for a company but was being done by someone who wasn’t a full-time employee at that company, like an intern or an interviewee. There were a number of r interesting technical and product challenges that they were trying to manage, and I just vibed with the group of people doing the work.

And in early 2020 I felt like an important part of that company. I could see not only the impact I was making, but vividly imagine a future where I could grow my career with the company and the people who worked there. We were on the cusp of signing some massive clients which could have fundamentally changed the scale we were working at.

At the time, I was working really hard with my team to deliver what felt like essential features.

And then the pandemic hit…

Within a few weeks, we went from being on the cusp of success to being on the brink of failure. We went through layoffs, pay decreases, and a remote working situation that made communication feel impossible. (Many of the tools and practices that we now take for granted really didn’t exist)

But, as a community, we pivoted. We found a new direction (within the travel space) to move as a company, and we started developing features to get there. It was weirdly a bit like summer camp for the final half of 2020. While we were developing features, there weren’t any users to test them, so there wasn’t that typical back and forth challenge of trying to prioritize maintaining existing code or shipping.

I even helped lead the architecture and launch one of the features I’m most proud of as an engineer. The beginnings of a financial matching and reconciliation system.

You might notice these are heavily rose-tinted lenses I’m looking back with. And… they absolutely are. I want to avoid papering over what were really challenging times or the arguments and back and forth that happened. But… the rose-tinting is important to frame my feelings. It was hard, but it felt purposeful.

And then… in early 2021 it happened. We got bought.

Joining Coupa

The funniest trope in acquisitions is how you’re still following “the same mission” but with “increased scope and resources”. It’s a silly statement, and I get why it gets said, but the reality is that new owners will have new and different opinions about what they would like to see. That’s… the point of an acquisition. It gets said because the alternative would be pretty crappy, “we’re going to change what you were working on and cared about, and you'll need to deal with it”.

The alternative, in an ideal world, also just isn’t entirely true. Ideally, both companies would like for there to be an on-ramp to that change. Maybe eventually your product changes, but there would be a slow wind down where your systems/tools/beliefs get absorbed into the larger org.

But getting acquired was a shell shock. Logically, I recognized trying to have maintained an independent small travel business through 2021 and 2022 would have been absolute madness. But emotionally, I was completely invested in the problem we were trying to solve and the way we were attempting to solve it.

In that sense, it was shattering. It wasn’t just that an external source had taken away something I (and my close coworkers) had been striving for. It wasn’t just that it felt so fundamentally out of the blue. It was that it felt like we had gotten so many of the things about working together as an org, about problem-solving, about communicating really right. The thought I kept coming back to was “oh, we got about as close as you could to getting it right, and we still were absolutely meaningless in the face of all of that”.

Trying to manage through those emotions while being promoted to engineering manager (a personal goal I’d had for many years in my career) while adjusting to a new company, culture, and responsibilities, was honestly just too much. I was actually burnt out. The dichotomy of getting the thing I had wanted (becoming a manager) while also having the identity I had attached it to ripped away (Pana Guest Travel) was a ton to process while also doing a job. So I left.

Joining Guild

In mid-2021 I left Coupa, took a 2-week break (where my wife and I roadtripped back to the West Coast) and joined Guild Education, a company that felt way closer to my heart. Guild had a mission of helping people I was fully bought in on, it aligned with a close personal interest of mine (education). I assumed that rediscovering that sort of emotional alignment, while working at a company that seemed to have an incredible employee experience, would, honestly, resolve my burnout. I had taken myself out of the fire.

And that’s when I realized I was completely and totally burnt out.

Usually when I start at a new company, I’m excited to jump in and start learning all the new things. Trying to get a better sense of what’s going on, how things work, and how I can start making a small but noticeable impact.

But at Guild, even though everyone was incredibly friendly, helpful, and patient, every time I tried to pick up new information I felt completely overwhelmed by the situation. I was confused by the terminology, by the flow of information, and by why everything that used to feel so natural felt so confusing. It was honestly completely disorienting.

So, I tried something new… I gave myself permission not to strive.

What Next

That’s all for this week! I wanted to focus on deep diving on the what and the why. Next week I’m going to dig into what recovery looked like, and why the Nike Running Club app became one of my go-to techniques.

This is continued in part 2 on recovering from burnout.

Recovery as a Process

I suspect for some, this will read like a typical story of self-rediscovery through mindfulness. I am skeptical of the idea that “getting more in touch with yourself” is the active driver of finding inner peace and happiness.

Instead, I believe that practice and habits are core to happiness and satisfaction. The rewarding elements of our lives aren’t the outcomes we achieve, but the challenges we face and the literal thing we practice and repeat. The act of practice gives space to let us deal with any number of mental/emotional/spiritual challenges, but for me, the practice isn’t a tool to encounter those things, it is something of value on its own. The practice is the thing.

Mental and Physical Injuries

The reason I spent so much time talking about physical injuries is because I think they are a useful comparison here. Even though there have been considerable strides in how we think about and treat mental health, we often fall into traps when we think about mental health and recovery. We think about diagnosis of the issues as half the battle (even though no one would say that about a physical injury), and we still bias towards personal blame instead of viewing behaviors through the lens of habits.

The reason I wanted to emphasize the strain analogy with burnout is that the way back from both are similar. To very slowly increase your effort without pushing yourself past your injury point. Like a hamstring strain that is easy to re-injure, burn out, is about finding ways to slowly increase your mental workload, without taking yourself back down. The other reason I liked the sports analogy is that there’s typically a set and knowable amount of time that an injury recovery will take: like 2–4 weeks to recover from an ankle sprain. What I found helpful about this metaphor is that it created a separation from my mental state and my personal identity. I could view my habits as symptoms of the burnout I was experiencing, instead of feeling bad. I could also set a period of time to say “it’s ok to take my foot off the gas”.

Healing

Taking the time off between jobs was a huge first step, but letting myself heal meant significantly resetting my personal and professional expectations. The second thing to note is that joining Guild did get me back to a baseline of stability.

This isn’t to say I stopped doing work. More like an arm in a cast, I set very specific limits on the types of work I would do, when I would stop, especially when I felt emotionally good, like I wanted to push harder.

There were a couple of times when this broke down, of course, and I did push harder than I intended. And in each of those cases, the exhaustion reared its ugly head again, and I took that as a sign that I needed to slow myself back down again.

I don’t remember where I read it, but I remember learning that burnout wasn’t caused by working for too hard for too long. I was caused by working towards something and feeling fundamentally out of control of the results.

This is honestly the biggest pain inflicted by burnout. The most exciting challenges are ones where the success isn’t directly tied to how hard you work. Instead of feeling like you had one “bad beat”, burnout makes you feel like any uncertain goal isn’t worth picking up, which makes it harder to fully participate in your life.

The Caveat

I’m writing this with the assumption that you, the reader, have the primary goal of recovering from burnout to get back to a similar place to where you were. It’s fully possible you want none of that. I make no judgments there, but I will suspect that you find this narrative journey a bit less helpful.

Starting to Run

I started running more often during the heigh of the pandemic, sometime around August 2020. The main reason, to be honest, is that I noticed how happy my dog Annie was whenever we ran, and when she was happy, I felt happy. And so, I started running, probably about 30 minutes a session, 3 times a week.

I decided to start training for a half-marathon because I think I saw an advertisement in 2021, and I felt like that would be a thing I could do. I think I stumbled my way into a path for success, to be honest. I don’t want to sound like I went into my burnout with a plan to get out. But there are some learnings that I have hypotheses about that I would like to share.

I don’t think mindfulness is a necessary component of this. While I happened to pick a practice associated with mindfulness, at no part was my goal on the journey to reconnect with myself. Like my analogy above, I think it’s very possible to be deeply connected to yourself and still experience burnout in a big way. So, here’s what I think is important to finding a burnout recovery habit:

1. It’s an activity I intrinsically enjoy, but succeeding at it wasn’t deeply attached to my identity
2. It had a goal I could brute-force
3. Guardrails
4. I had a set end goal
5. I gave myself permission to prioritize it
6. It was challenging

1. I Enjoy Running, But it’s not connected to my Identity

I love running. The act of running makes me feel great. But I am not a fast runner. My enjoyment of running is not, in any way, shape, or form, impacted by how fast I am.

This was actually quite key. Because early on, one of the hobbies I wondered about picking up was something like game design. Why not go back to something I loved that gave me a lot of meaning?

What I realized was, my identity around game design was tied up in my ability to be good at it, so that same sort of striving and judgment would creep in. And this isn’t to imply that I’m a competent game designer. I am deeply not. But I had an emotional attachment to the idea of being good at game design in a way that I wouldn’t as a runner.

2. A Brute-Forceable Goal

Brute-Force is a software concept where your ability to solve a problem is solely based on a factor of time. To use running as an example, this means that my goal was to finish a half-marathon, it was not to set a certain time at said half-marathon.

This is also why I might encourage not focusing on a creative pursuit (where creativity is the goal). To go back to the game design example, even something as simple as “finish a game” doesn’t have a clear outcome. It requires me to either listen to feedback or to spend time trying to come up with a set goal.

Running a longer distance is the type of thing I know I can do simply by running more. I might not run well or run longer faster, but I know if I spend enough time at it, I can increase the distances I run at.

I think other pursuits could absolutely fit in these cases, both physical and more intellectual. Maybe it’s hitting a number of free throws (not a percentage), or sketching some number of pieces of art, or throwing some number of clay bowls. But I truly believe quality should not be part of the success equation. The goal was to disconnect my identity from success in the challenge.

3. Guardrails

The other thing I’ll note is that I leaned heavily onto the Nike Running Club app. Honestly, I’m not entirely sure how good it is, I’m not an athlete. But what I do know is that I didn’t have to think about a training plan. I just pressed play. That sort of divorcing myself from the goal entirely again meant I could just focus on the act of doing.

This created a number of self-reinforcing behaviors. Because I didn’t have to think about what I was doing I could pick it up quicker, there were fewer veto points for me to say no or chicken out.

I think if I were to try something similar without those guardrails (like maybe drawing), I would probably focus on trying to make a medium/long-term contract with myself where I would just pick a thing and do it. Potentially, I would practice drawing the same hallway for a month. That sort of thing seems inane, but the goal here is setting goals early that eliminates thinking in the future.

4. There Was a Set End Goal

I had a date and a goal in mind. This connects to the concept of an injury because I wasn’t trying to change my identity or become something new. I had a specific goal with a specific timespan in mind. Once that date came, I could renegotiate my relationship to running or drop it entirely.

5. I gave myself permission to prioritize it

This is one of those places where the privilege of having a flexible, remote job absolutely come into play, though I suspect there are other ways to make time. The point I want to emphasize is that there were times when running came into conflict with other priorities in my life, especially ones that I connected to burnout. Specifically, those times when I might’ve started working early or stayed late, I ran instead. It was honestly hard to do so because being a hard worker is intertwined with how I view myself, but I had made the agreement that the path to heal wasn’t through working more, it was through running.

6. It Challenged Me

The key here was that running still challenged me. There are so many times when I started an 8-mile run, internally screaming at how annoyed I was to be outside running. Or 4 miles in, begging myself to stop. So, it’s not like it was easy. But the easy part was that I only had one decision to make… either keep going or stop.

But… I also think it’s somewhat different from the meditate and rest form of recovery we talk about, and worth talking about the differences. When I was completely burnt out at the end of a previous job, literally taking the time off was important. Resting and disconnecting were akin to letting the sprain heal. To reach my goals again to get myself back on the path I wanted, active recovery was necessary. The challenge of it wasn’t a side effect of the goal, it was part of the recovery process.

What Actually Happened

In October 2021, I ran my first half-marathon and finished. It was around that time I started taking on more challenging projects at work again, and started feeling like I could participate fully in work, even knowing the uncertainty that my efforts would be fully rewarded.

What this practice of running taught me, is that it rebuilt my confidence in my ability to face challenge and not immediately assume the outcome was meaningless. It taught me that I could enjoy the thing for the sake of the challenge, without the desire to hit specific metrics or to be the best.

It also taught me that I could be flexible and renegotiate the things that were most important to me.

Renegotiating my Concept of Success

Like all injuries, sometimes it’s not just about healing back to 100%, it’s about renegotiating your relationship to the reality you live within. I value stability much more than I did, I’m much more skeptical of a crunching lifestyle and of the concept that that crunch will be rewarded by much of anything. My relationship to my job, even though I still strive, has changed dramatically. It would be weird if I went through this and was fundamentally the same person. I truly believe the specific running habit I practiced in late 2021 was a significant part of that recovery. But I don’t believe it has to be running, or even physical activity. The principles of having an enjoyable activity, disconnected from my identity, with a specific goal that could be achieved with effort and structure, can benefit anyone looking to recover from burnout.

The over-justification effect, which, I believe, is directly connected to burnout, is the concept that as we focus increasingly on extrinsic rewards the intrinsic values we found disappear.

Burnout is, in my opinion, the harshest form of over-justification. We become so focused on that external reward that when it disappears, there is nothing intrinsic left to come back to. In those situations, we cannot simply remind ourselves that once upon a time, we had intrinsic rewards from a certain behavior.

It can be hard to recognize that we can see how things once were, and also that our seeing it doesn’t make it possible that we can just mentally force ourselves back there. Just like physical changes, mental changes take time and effort to change. But, it’s possible to step away from that part of ourselves and find new areas for discovery.

You should make a habit of giving someone positive, formal, written feedback. Make sure their manager sees it as well. I started doing this close to a year ago now, as a way to try to give back to my coworkers and I think on a personal and professional level. And I’ve just kept going because every time I do it, I feel more content with my job.

If you’re reading this, it’s likely that you have some sort of formal feedback tracking system that you can input. And if you’re not, you can send an email.

1. The Selfish Reason

Writing lots of feedback, I believe, is an important point of visibility in your favor. As you share feedback with managers, it spreads your influence, shows that you care about (and can speak to) company values, and puts you as someone they’re connected to in your head. It can be incredibly hard to get into “the conversation”, but sharing feedback is a reasonable, low effort way to put yourself in front of management without having to find “essential projects” or “sponsors”. It also creates a better relationship for you with the person you just had an interaction with, and it makes them more likely to give you feedback in the future.

2. The Altruistic Reason

This is definitely the coworker version of “tell your friends and family you love them”. There’s never been a point in my career where I’ve felt I was getting too much (positive or negative) feedback. It’s so easy to assume that because you enjoy working with your coworkers, that they know your positive feelings. They probably don't, and even if they do, you should take a few minutes to put it in writing. It's a good reminder, and helps them get visibility on their positive work with their managers.

Most feedback that gets shared, in my experience, is the big stuff. A project that went well, a raise, or a project that went poorly, or some big feedback about behavior that needed to be corrected. The small stuff, the clarifying, steadying stuff often goes unnoticed.

3. The Semi-altruistic Reason

If you’re at a large company, most feedback is going to come in waves at the one (or maybe two) performance cycles that happen on a yearly basis. If you’re only providing feedback at those times, it’s more likely that your feedback will be lumped in as a generic “positive”. When you share the feedback outside those times, it’s more likely the specifics get recognized and remembered. This means their manager will take note (a benefit to them) and your feedback will be remembered and acted on.

In Conclusion

A stained glass window of someone smiling at a letter

It takes about 5 minutes to do, just open up your email or Lattice or 15Five or ADP or whatever you use, find someone who you interacted with this week, and write them something. You’ll feel good and they’ll appreciate it. It’ll make your place of work the tiniest bit better, coworkers feel a bit more seen, and make you feel good.

So go do it. Open your tool of choice and drop someone a quick bit of positive feedback.

I remember learning about test-driven development or observability as a strategy, primarily as ways to improve the trust and reliability of my code. But as a junior developer, I remember how much those things felt like restrictions rather than empowerments. I remember thinking about how silly it was to be asked to do all of this extra work or configuration to ship the same amount of functionality.

On reflection, this is quite hilarious to me. React has basically been sold as a Developer Experience benefit, even in the early days when it required an obscene amount of Webpack setup, the benefit was always better DX. And the same for Typescript. But no one ever said TDD was a better DX, they just said I had to if I wanted to call my code good, so I rebelled against it.

So, I want to take a second to talk about the benefits of testing and observability, through the lens of the

How do you know your code works?

But as I got more experience, and as I started joining more teams, I began to see the benefits of both of the things. There’s nothing quite like the experience of going through your first on-call rotation and experiencing waves of bugs with little to know warning and having to try to figure out what the system is doing. It really sucks. And then comes the slack journey of trying to figure out which PM or dev wrote the tickets or the code. Once you’ve found the dev/pm, you can attempt to figure out if the code is actually working the way they intended, often within a tight and stressful timeline.

There’s a lot of bike shedding around what encapsulate test-driven development and what good observability looks like, and how to tell if you’re following “best practices” at any of these things. I don’t find this critique focused method of analysis a particularly great way to encourage myself or others to jump onto the testing or observability bandwagons, or particularly good internal heuristics for understanding if I’m doing it well.

So instead, I’d like to posit we think about tests and observability in a different way: we should think about them as ways of building our internal image of the code.

Seeing With Code

The first class I took in college was Observational Drawing. My art teacher was a soft-spoken, critical man with a belief that anyone could learn to draw because drawing wasn’t about talent, it was about changing how you see the world. He would say we spend so much time looking through our eyes that our brains would just start to abstract shapes into ideas. So instead of seeing collections of lines and shadows, we would see a house or a person. Learning to draw was about stepping back and seeing those lines and shapes and shadows again, instead of the idea. He would encourage us to hold up our pencils against the object we were trying to recreate, so we could get a real idea of the angle or the size or the distance. Basically, a way of forcing our brains to deal with the world as it is instead of the abstraction.

I think of my growth with software engineering similarly. I spent so much of my time thinking about getting to the execution of the whole that I would often miss the small building blocks to getting there.

My Limited Perspective

Starting in bootcamp as a developer, the first way I “saw” my code was by running it on my personal machine. I would make a change and either run it in the terminal, open up a browser, or fire off a request and see if it “did the thing” I expected. If it didn’t, I would debug the issue, make some modifications to my code, and then try again. It’s sort of the “guess and check” methodology.

Seeing Through Feedback

As I got more experienced, code reviews became part of that practice, and then sending the code off to a PM or a tester to exercise it and give me feedback. So, my view of the code expanded. I didn’t just think about whether I could execute the code, but whether I could hand it off, and it would succeed even if others were doing the testing.

But these ways of seeing and visualizing code are quite self limiting. You only have to go through a few rounds of on-call and a few incidents to understand just how quickly even chaining 2 to 3 functions together can create a plethora of outcomes you didn’t intend.

Seeing With Tests

That’s why the next level of seeing, is seeing with tests. I fundamentally believe that unit tests and functional tests don’t primarily exist to “protect” our code from future bugs. Unit tests and functional tests exist to encapsulate the boundaries of the functions and services we write. They help us understand, articulate, and see our code better, and they help us share that knowledge with other developers.

When you make a change to a piece of well-tested code and a unit test starts failing, it’s not necessarily true that that test prevented a bug from going out. But it is true that the unit test failure helped the other developer (maybe future you) have a clearer picture of how the developer who wrote or maintained it was expecting it to operate. It creates a picture that’s broader than the scope of a single execution and starts helping us paint the boundaries of the system. As a result, we get stabler code, but we also get faster development and a better development experience. Thinking about tests only as a way of catching “bad stuff” is self limiting and will make the tests you write less helpful (and less likely to catch the bad stuff).

If tests are step one, the next phase of this is code that’s made it out to production. Tests are still limited by your imperfect understanding of how your functionality might work. As you scale, you’re still dependent on pings and pongs from other users/teams to tell you whether stuff’s going wrong.

Now when I ship code, if there aren’t tests associated with it, my default assumption is that I don’t have a particularly good understanding of what that code does. It’s a quick sketch at best.

Seeing with Observability*

* I realize reading this as a “normal person” seems rather obvious. But in technical terms, observability means using some sort of tool like DataDog or Chronosphere to introspect what your code is doing.

Monitors, like tests, are a terrifying level of abstract that requires a similar shift in how we think about our code. Instead of seeing it as a user experience or even a set of requirements, now we see it as a set of numbers, lines, and dots on a screen. We can see things like execution time, errors, sequel queries. It’s pulling us away from the direct user experience which we’re comfortable with and towards further abstractions. But just like lines and shadows, these abstractions help get us closer as engineers to maintaining and building solid codebases.

I’ve started trying this method that I call alerts that cause joy. It can seem absolutely wild to think that an alert could cause joy, but its fundamental purpose is to support developers. As a developer, you basically have two options:

1. You can wait until someone tells you something has gone wrong. Potentially to many people.
2. You can be told immediately, and have an opportunity to fix it before it gets bad.

There’s nothing worse than being on a support cycle and showing up Monday morning to 15 bugs related to two issues that you now have to parse your way through.

Waiting for user reports to let you know something has gone wrong is the worst. And if you don’t have observability in place, that is absolutely where you are. An observability system doesn’t just tell you when things go wrong. It gives you a window into how your system is operating. Similar to testing, if you think about monitoring only as the process of catching errors, you’re self limiting. Observability systems can show you an entire scope of your application and give you an early indication before things might go wrong.

Now if I ship code and don’t have the ability to look at traces, metrics, and logs together, I assume I basically don’t know what’s happening to it.

Changing How We Write Code

DALL-E rendering of the ui of a monitoring system that points to the exact location in a codebase where a bug is

The reason I want to compare tests and observability to drawing is because at the end of the day, our goal in both is to accomplish the objective. To create a cohesive experience for our users. And we still have to keep our eyes on that level of understanding. It doesn’t matter whether we draw the best lines or write the best tests if the whole isn’t coherent and meaningful. But what the techniques of observation and testing do is help give us tools to improve the scope of our sight, to create a richer experience for a wider variety of people.

A former boss came up with an excellent metaphor for this sort of development, sick is worse than dead. It’s the concept that in a pre-testing and pre-monitoring mindset, we’ll end up with functions that handle a plethora of use cases without really failing, which means users don’t see errors. But they also won’t really succeed, which leaves devs with a lot of work to suss out what’s going on and to try to fix it.

Adding Tests

Often, when we start writing tests against a mature codebase, it can feel incredibly painful. Flexible functions with a variety of outputs are incredibly difficult to test. If the shape of an object in a response can vary based on the types of input, the optimal outcome is to write tests for each of those use cases. But enumerating them out is so hard.

Tests actually require that we start writing different types of functions that operate in a much smaller scope, so we can use tests to explain and understand them. And this can feel strange, uncomfortable and limiting because we can’t just write a thing to do a thing, we have to write a function that can be tested.

Tests are limiting in that way because they mean the code we write has a much narrower scope. But within that scope, we have a higher trust that what it’s telling us is something it truly believes.

Adding Observability

I’ve seen numerous times when instead of letting a request fail, the code will catch the error, log it and then move on. The hope is that the user wasn’t impacted, and a dev can go back and clean the issue. (in our imaginations) But in practice, these errors rarely get resolved. It’s because handling errors this way kneecaps our observability systems’ abilities to actually tell us what’s going on and makes it harder for us as developers to improve our systems over time.

Like tests, this can feel strange, limiting and uncomfortable. Instead of letting a user simply continue on, we’re increasing the chance that they’ll get blocked and that it will be an immediate negative experience. But the tradeoff is that by letting these issues surface we can catch them more quickly, before they impact more people, and before they can spiral out to downstream issues causing challenges that are increasingly hard to resolve meaningfully.

Systems that are built to be monitored have better client contracts, better developer experiences, and ultimately a better user experience. The developers who support these systems have a much clearer image of the code they’re trying to support.

Back to Baggage Claims

Part of the :reason I find the metaphor for baggage systems in airports so helpful is that unlike websites, you would never try to analyze the health of the system based on simply attempting to send one bag through. At best, it’s an incredibly limited way of seeing what’s going on in one use case, and at worst it gives you a biased perspective on what your users experience.

That same boss came up with another gem, good observability gives you the same understanding at scale as you have early days when you’re first spinning up an application on your personal laptop.

Pulling up to a more abstract level can feel like letting go of your previous ways of seeing the world that were more concrete and more real. But the tools are in place to help us write code that gives us better opportunities to create a great experience for our users.

I appreciate anyone who can take on what being a "Staff Engineer" means. It broadly means you're "important" (either through earned value or title inflation), and it also means you're "different" than other engineers in some specific way (either through earned value or title inflation). So the role is weird and I appreciate anyone who tries to take it on in a thorough and nuanced way. I found it to be at once a thoroughly researched and excellently sourced reference, an overwhelming tome, and a snappy read with a number of easy to apply insights.

I found it to be at once a thoroughly researched and excellently sourced reference, an overwhelming tome, and a snappy read with a number of easy to apply insights. I’m going to try to break it out into the different components that I found the most salient, by grouping them into affirming, novel, expansive, and challenging to implement. This is meant as less of a direct critique and more of a "here's how I'm thinking about the book". My assumption is that a good portion of this is influenced based on how relatively new as a staff engineer I am. I suspect for some readers the things I found insightful might seem rather obvious, and that's ok!

The Principles I Found Most Impactful

• :Ask obvious questions, make the implicit explicit
• :Make Sure the People Around Help You Raise Your Game
• :Celebrate landings not launches
• :Making work manageable for someone else
• :Gently direct your colleagues to the most valuable work
• :There are different level of staff+ engineering reporting
• :What can you do to make 14,000 engineers better?
• :Creating Maps of Your Org
• :There are different types of staff engineering

Affirming

These were things that I already knew (or at least already thought I knew) but it felt good to hear them from someone who has much more experience than I do.

Ask obvious questions, make the implicit explicit

This is a skill I think I’m particularly good at, and it was nice to hear that it’s something that might be considered valuable as a staff engineer. The point Tanya makes about these is that often there’ll be a ton of implicit assumptions about the way things are. Rather than letting those lie, it’s important to jump in and get them documented so that they can be understood, and potentially challenged. She says, “It’s better to be wrong than vague” and it's good advice. It's way better to say something specific and be told you're wrong, so your team can succeed, than to say something vague, technically not be wrong, and have your team languish.

Make Sure the People Around Help You Raise Your Game

One thing I genuinely struggle with is occasionally being selfish. One of the points Tanya comes back to is you can’t be effective as a staff engineer unless you pay attention to your scope, influence, and happiness. Having developers around you that help you raise your game and push you to do better work is critical. It's critical both for your future growth and also to make sure you stay active and engaged in your role. And it was affirming to hear, to reassure me that I can look out for myself occasionally.

Expansive

These are things that added to and extended my previous understanding of the world. They might’ve been things I had intuitions around but hadn’t taken to the next step.

Celebrate landings not launches

This had been a direction my perspectives have been evolving over the last few years anyway, the idea that your code being stable over a long period of time, on production, is equal to shipping the code. But “celebrating landings” helped solidify that concept. It’s the idea that you should actually wait (or in my case additionally) celebrate when the old code is gone, when you’re comfortably stable, not just when you get the thing out there. It’s something I plan to do with teams I’m on moving forward.

Make work manageable for someone else

As a staff engineer, starting out, it was hard to understand where my work begins and ends. As Tanya repeats throughout the book, if I’m working on code, it’s likely at the expense of someone else’s experience. But finding limits around trying to wake complex work more manageable for other engineers creates better boundaries for how I can be impactful without harming someone else’s potential learning. It’s imperfect, but I have pretty good senses of places where I think other developers might be overwhelmed, and I can also ask them about it. If I think it might be too overwhelming, I can help out. But If I think it’s enough uncertainty, I don’t need to step in, and I can let them learn through experience.

Gently direct your colleagues to the most valuable work

This is one of those ideas that connects to general concepts of leadership without authority I’ve had for a while. But this makes it more concrete, in the sense that you should use your insight to help direct teammates towards work that you think will have a good impact on their career. This is the corollary to taking on “glue work” that might be important but not showy (that staffs should take on). Staffs should also encourage other devs to take on the sorts of high profile, impactful work that can make a difference in their career.

A Dall-E rendering of: A pencil sketch of someone looking at a map of a road

Novel

These are concepts I found genuinely new or exciting, putting together ideas I hadn't even considered previously.

There are different levels of staff+ engineering reporting

I really appreciated Tanya calling out the varying levels of reporting that a Staff+ Engineer could direct at. From a line manager all the way up to a VP. In one discussion I had with a former boss, he thought this sort of progression helped show that individual experience scaled up equivalently with management growth. But, I found it helpful to understand how reporting low and high has different benefits and can change the way you interact with an organization, especially as it relates to your personal and professional goals.

She also assigned the words “reporting high” and “reporting low” for the various ways of thinking about these things. It was really helpful to get a picture of the pros and cons of each level of reporting. The higher you get, the more expansive your scope but the fuzzier your view, and the lower you get, the narrower your scope but the more sharp your view.

What can you do to make 14,000 engineers better?

The specific number, 14,000, came from a quote from a principal engineer in the book, but the point is the same regardless of the multiplier. I don’t want to admit that this one blew my mind, but it did a bit. The point, as she discusses at multiple points through the book, is this idea that you have to think about how you can make an impact on more than just one person at a time. And being explicit about that scale. Whether it’s a team of 5 devs, or teams of 25 devs, that scale is bigger than what anyone on one or even one on two interactions can manage. And finding ways to think about impacting multiple engineers is something I’m going to sit with for a while.

Challenging to Implement

These are ideas I found intriguing, and potentially beneficial, but acting on them felt too time-consuming or complex.

Creating Maps of Your Org

The book highlights three types of maps you can make, as a staff engineer, to help give yourself a better understanding of where you sit and how you can be impactful:

• Treasure
• Locator
• Topographical

These are different ways of seeing your employer and where you sit within it. They’re supposed to help situate you inside your organization and make you more effective. The different views, which remind me a bit of Jesse Schell’s 100 lenses of game design, were about your personal goals and the various relationships between different teams.

I really appreciated the focus on different ways of seeing your org, but actually implementing these maps felt overwhelming and close to impossible. If I had a free week I could probably sketch out it all, I might be able to articulate something, but I’m not actually sure how useful such a map would be. I wished there were more directly implementable ideas I could jump into with a bit less effort.

There are different types of staff engineering

The book outlines 4 different types of staff engineering work (show below) to highlight the different ways that a staff+ engineer can engage and drive a project forward.

• Core Technical Skills
• Product Management
• Project Management
• People Management

Tanya calls out the different flavors of work that a staff engineer can take on. The point she was trying to make, I think, is that there’s a variety of different behaviors that you can (and maybe should?) flex into across your career depending on where the moment takes you. But, while each of these certainly feels like types of work I’ve done in the past, whether that work is valuable is heavily constrained by the organization you happen to be in. I’ve had managers at various jobs explain to me that engineers really shouldn’t be doing any product work to create a clearer separation of responsibility. If that’s the case, what does that mean, and how should I be thinking about these flavors of staff engineering? The book left me a bit wanting more in that regard.

A Good Resource

If I had a recommendation, I would probably say similar to Camille Fournier’s The Manager’s Path, it’s probably best to take a section or a couple of chapters at a time that feel most applicable to you and your role, and focus on those, instead of trying to read it all in one go. But the writing, level of research, and insight is consistent throughout the book, and it’s one a plan to come back to from time to time.

Epilogue: Sections I Would Focus on

If you have checked the book out, or on loan from friend, here's what I would focus on, ranked:

• Chapter 3: Creating the Big Picture (Pages 69-111)
• Chapter 8: Good Influence at Scale (253 - 284)
• Chapter 1: What Would You Say You Do Here (Pages 3-30)
• Chapter 4: Finite Time (115 - 150)

And what’s weird about leaving a team is that it’s one of the few times when our emotional responses to situations put us out of sync with what is best for us and for others around us. I think that, for the most part, what feels like the right decision during your employment will be the appropriate decision. There are management hacks, tips, and tricks to nudge ourselves to be more efficient, but mostly how we feel and what we should do are pretty well aligned.

During times when we’re preparing to leave a team or a collaboration, our emotions can upend our behavior that can make our own incentives misaligned with the incentives of our team and management. This dichotomy is what makes it hard to understand how to leave a team well. Coming to terms with the challenges will help you leave a more lasting impact, and make it more likely that they appreciate the work you did and how you left.

There are three responses during this process that I’m going to focus on:

• Things You Need to Come to Terms With
• Things You Probably Care About, But Shouldn’t
• Things To Finish As You Leave

Things to Come to Terms With

You Just Have Like… Way Less Time Than You Think.

Deadlines create emotional incentives for us to complete stuff we’d been putting off for long periods of time. And exiting is a huge, emotional deadline. But leaving a team is not the right time to check off a bunch of stuff on your checklist. It is the appropriate time to prioritize even more aggressively and recognize you’ve got one, maybe two things you can hand off really well, and make sure your efforts are focused there.

Finishing projects on time is hard enough in perfect circumstances. Finishing projects while you’re also doing the thing where you’re trying to unwind your existing responsibilities. The nature of all human endeavor is that you’ll always be covering areas that aren’t exactly articulate, or you might have priorities, goals, and beliefs that you would like to see continued after you leave. Emotionally for you, and practically for your team, it’s better that you take stock of these and figure out if there are one or two that you can hand off. Don’t try to wrap up them up in a neat bow.

Leaving Knowledge gaps

Your goal with creating a successful rolloff is not to create 15 pages of documentation. Creating extensive documentation right before you leave probably isn’t all that helpful. In part because a “brain dump” perspective isn’t conducive to good writing, and in part because you’re not going to be around to explain any new ideas that come up. Imagine you were going to school and your teacher was going to start teaching a different class that week and tried to cram everything they might have taught you for the quarter in that week. It would suck.

Instead, focus on making sure the initiatives or projects you had ownership of have people who will take care of them. Do this at the start of your rolloff and make sure you’re around for teammates to pull information from, but don’t worry about the stuff that’s going to drop. It’s not your job to figure it out, and even if it were, you don’t have enough time to do it well.

Your unfinished work or ongoing commitments

You might have an agreement where your relationship to a given team is contingent on the completion of a project and a set of projects. (Maybe you’re a consultant working in an area of the division, or maybe you’ve been an embedded resource) But unless there are very specific circumstances (and I would argue even in those cases there will probably be some misses) you’re not going to finish them.

Your goal in your remaining time is absolutely not to finish them. There is no world where you finish your commitments, leave gracefully, and are recognized for the awesomeness of those commitments. So just… stop. The more likely outcome is that you make a royal mess, under-communicate and underdeliver and everyone else will be less happy as a result, including you, who was stressed right until the last moment.

The first time I ever left a job as a dev, I was so focused on finishing this big project because I really wanted them to remember me well. I thought if I could deliver this one thing, I would be remembered really well. It was part of a service that I had built out and owned, and I wanted to make sure it was perfect before I left.

Flash forward months later… and chatting with old coworkers, they let me know that the work I thought I crushed was full of bugs. They didn’t really understand what I had accomplished, and it caused a lot more problems than it was worth.

Things Not to Care About

Your professional reputation, as it relates to the projects you’ve been working on

As a corollary to not caring about previous work, don’t let your desire “leave professionally” cause you to do too much or prioritize incorrectly. There will be gaps when you leave, there will be areas that people will express frustration about how you left if it causes them frustration. I don’t think it’s worth trying to make an extra effort to prevent those frustrations.

My experiences have led me to the conclusion that the second your departure has been announced, the people on your team are already planning and thinking about life without you. They’re already thinking about you in the past tense. This isn’t a bad thing, in fact, it should take some pressure off because how they will feel about you is mostly already baked in. Just be the same great person you were leading up to the move, and your professional relationship will probably remain the same.

Thinking about how your departure will impact team dynamics

Maybe I’m just super self-centered, but whenever I’ve left or changed teams, I’ve always imagined how my departure might impact the team I’ve been on. I’ve wondered how they’ll fill the gap, and imagine what might accidentally blow up.

The reality is, that either you’ve done a good enough job communicating and partnering with team members pre-move that they’ll be happy to continue championing your goals post-leave, or you haven’t and they won’t. So leaving isn’t the right time to try to do that communication.

Your priorities aren’t the team's priorities (even when you were there). This falls pretty closely in line with Let It Fail, where the reality is that if you were heroically holding up some aging system, you probably weren’t doing your team or yourself any favors. Your leaving is most likely not going to cause an immediate collapse, but is much more likely to cause some pain that either gets resolved or carries on. It’s a much less exciting outcome (especially if you, like me, have an attachment to feeling important), but it’s the truth.

While this might sound depressing, it should also be freeing. You don't need to take herculean action in your last 2 weeks, just point out potential gaps and let the people that remain take care of it. Instead of over documenting or fretting, use the two weeks to emotionally let go of those things you’ve cared about. Talk through the situation with your coworkers and management, and support them in the situations where they ask for help, but don’t feel the need to push all of your knowledge out. The team will find a way.

Things to Finish

Assign owners for remaining tasks to make sure that they get picked up

Just because the work you do in the last two weeks is ephemeral doesn’t mean you have no responsibilities. Leaving well means making sure that critical tasks you were on have owners and that those owners know how to reach out to you while you’re still there. Do this early and often. Do Not worry about boring them with a checklist of every small thing in your head. Explain enough that they can bear the weight of the added responsibility, say you’re around, and then let them handle the solution. They will take things in a different direction than you might have, or they might let it drop entirely, but post-hand-off that is their decision that they can make one way or another. Your team will appreciate your having done so.

The logistics of leaving

Focus on the project management of it all. Instead of worrying about the stuff that will continue on without you, focus on the very practical stuff, like the timeline of your exit. When will your move be formalized? What Slack channels will you remain in (and for how long) and what will you exit? When will that happen? Are there meetings you’re going to stay attached to? For the rest, when do you intend to stop showing up?

Are there specific meetings you want to have to formally hand off work? Are there launches or other deadlines that will occur within your final weeks of being on that team? How do you plan to handle those?

I would recommend spending a bunch of time thinking about the dates and times that various moments have to happen, and getting good agreements with the leaders on your team as to why and when they’ll occur. That way you have a clear plan.

Be Done Early

The one job I felt I had a semi-successful exit from, I was done a week early and was sitting on my hands. I felt really silly for doing this, but in retrospect it was actually my most successful exit. Like I was taking up unnecessary space. But what it actually meant is that there was enough slack in the system for the team not to have the pressure of my leaving. I was around to answer any questions, but wasn’t critical to having those questions answered. It’s an example of Celebrating a Successful Landing. The success wasn’t that a bunch of stuff got wrapped up, the success was that the transition occurred without lots of upheaval or chaos.

Leaving Is Hard

A Dall-E rendering of an oil painting of a man waving goodbye to close personal friends to go an exciting journey with a beautiful mountain in the background

No matter how you spin it, doing any sort of move from a project or team you were on to another one is hard. Even if it’s within the same company, as the result of a successful outcome, and you’re excited about the future, there’s a whole host of emotions that comes along with it. Give yourself the time and space to process those emotions, as part of the leaving process, and not as something to deal with after it’s all over.

I was supposed to spend this week writing about confidence and certainty, but then I got sick and that got sidetracked. I’ve got mentorship stuff on my brain, so I wanted to string some words together about it.

When The Last Jedi came out, I never stopped thinking about the scene between Luke and Yoda and the Jedi temple. Besides the fact that Yoda destroys some Jedi trees with forcing lightning in one of the best representations of chaotic good I’ve ever seen, it’s a poignant meditation on mentorship. It expresses the fact that you never really stop growing, and you never really stop feeling nervous or uncertain about yourself. Luke’s like the most talented Jedi in the galaxy, and he still needs guidance.

That resonates forcefully with my experience leveling up as an engineer. You’re so excited to get to the next level to escape your insecurities, only to find out that the insecurities come with you, they just change shape.

Leading up to this, Luke:

• Over assumes how Rey will behave based on his experience
• Views himself as a failure for being capable of teaching her the ways of the Jedi
• Stays stuck within historical ways of seeing the world through the lens of Jedi teaching

And Yoda comes to comfort him and to take the burden of that responsibility off him. But he doesn’t do it with a pat on the shoulder or a “chin up, you must keep”. He gets at the heart of the flaws, limitations, and power of being a teacher.

“We are what they grow beyond”.

I literally have never forgotten that line. It encapsulates the beauty, optimism, and anxieties of the mentor/mentee relationship, while also gently nudging about how to think about being a mentor. Sometimes Rian Johnson just freaking gets it.

I want to reflect on some lessons on it, not because I think you’re going to like… get a promotion if you listen to Yoda or something stupid. I think it’s worth chatting through because the lessons will make you a better mentor and have more fulfilling relationships at work.

The Fear Never Goes Away

One of the most comforting parts about the scene, and something I’ve experienced a lot during mentorship, is that the fear never goes away. When I was starting out as a dev, I imagined becoming increasingly important and skilled and that reducing the amount of fear and uncertainty I dealt with. I didn’t know if I was coding the right way, or if I would accidentally do something that caused chaos or would get me fired. I am definitely less worried about those specific things. I’m confident in my ability to do stuff with code.

Boom, now I’m a Staff Engineer and I still face nervousness. But now I can feel the limitation of my viewpoint in a way I couldn’t at the start of my career, and that makes me act more cautiously. I can see the potential to use my position to accidentally constrain the thinking of the people around me. And yes, I’m occasionally worried about being lapped by the people I’m supposed to mentor.

The Growth Beyond

Especially in professional contexts, it’s easy to see the mentor/mentee relationship from the lens of the potential to level up further via roles, responsibilities, and pay. That encodes the mentor-mentee relationship in this sort of one dimensional hierarchy where the mentor is like a tugboat slowly pulling their mentee along behind them. We see this in the Star Wars universe with the growth from Padawan, to Apprentice, to Knight, to Master.

But Yoda’s point explodes the entire idea of teaching being one dimensional. Mentorship helps mentees grow to see the world in ways that you hadn’t, and that will push them past your understanding. And in doing so, they can then teach you. It’s one of the few places where fresh eyes can be informed by experience without being constrained by it. And instead, use that experience as a stepping stone to strive further.

That’s the magic, that’s the “force”, whatever you want to call it, that’s the cool thing that happens when you teach and learn. It’s not a dry knowledge transfer, it’s a multiplicative relationship that leaves both people better off.

Your Limitations Aren’t Theirs, and that’s ok

But that’s what makes the point of what Yoda does so beautiful (and what that phrase encapsulates). The mentor-mentee relationship isn’t about the limitations of one party or another. It’s not about the comparisons or the anxieties. It’s about the lenses that we get with age and experience, and being able to impart those lenses in ways that help our mentees accomplish wonderful things.

You don’t have to be free of doubt, uncertainty, or your own mental limitations to mentor someone else. You just have to be honest, open, and communicative. And trusting, and believing that they will get what they need from you, with the right attention and care.

Seeing it Happen Is So Cool

There is nothing more exciting that watching someone you've been teaching apply the concepts you've taught and then turn around and extend them in ways you hadn't even considered. It is at once humbling, invigorating, and exhilarating.

From Dall-E: a visual representation of the transfer of knowledge between a teacher and a pupil rendered like concept art from the original star wars

The line is also a good north star. It asks whether you're behaving like someone who is pushing the people you interact with beyond your limitations, or if you're holding them captive and reinforcing their need for you to be around. In that way it's a vision of servant leadership that is sincere and not bullshit.

The Human Side of Work Matters

I get so frustrated with LinkedIn speak sometime. Where concepts about leadership and ethics get flattened into these weird binary optimizations. These are the 5 best ways to be a mentor or, here are 5 takeaways for how mentorship can help your career. Yeah, sure, it can.

But, to flatten it down to something so simple is insulting. Mentorship is fraught, and challenging, and can be done wrong. The benefits of mentorship aren’t a flip you can switch, and there’s no guarantee that your mentee will get anything out of what you give them. And it can be very challenging to grapple with that idea.

But, mentorship is so rewarding is that it enriches our lives in ways that are challenging and personally meaningful, regardless of the career outcome. It really is magic. It takes all the knowledge that you’ve built up, and it doesn’t just hand it to someone else in some sort of additive transaction. It can slingshot that person, and in doing so, lift everyone around you. It will also enrich your lives in the way that gives it lasting meaning, beyond the daily positive and negative outcomes.

In one of my last :articles, I argued that you should give feedback (in part) because it would be good for your career. I don’t think you should think about your career when you mentor. You should mentor because it’s a rare magic, and good, and beneficial for society if you try to do it more, even if you occasionally fail.

cartoon bubbles of two people communicating like in calvin and hobbes

1. Will engineers continue to use open-source code packages heavily?

I’m imagining something like lodash where, the package is open source, publicly accessible, and something that’s easy for ChatGPT to spit back out. Now that you can just ask it to spit out something like a sort with full unit tests, is there any value to adding lodash? Or will people just start re-using bespoke utilities and not installing the full set of requirements. It’s harder to imagine something like React going away, but smaller utils that do more specific things, one could imagine disappearing entirely.

2. How much will it reduce human to human communication?

It is theoretically possible that instead of asking anyone about anything else, you could simply query ChatGPT for any context you need from a coworker. Will we see situations where people stop communicating with one another altogether? Or will we communicate to one another about different types of things? How will that impact things like team psychological safety? Where communication is a core part of the way safety gets built. This is one of those things that strikes home because human to human communication is the sort of thing that’s filled with friction that we often like to avoid if we can. But, when we abstract it away, it can reduce our trust in one another overall and create bigger challenges as a team and company.

3. How and where will the hallucination-tax strike?

Chat-GPT hallucinates facts. And while, it does great at synthesizing things, you can’t actually validate whether the hallucination is real unless you actually read the underlying material. In the case of summaries, bullet points, and the like, reading the underlying material defeats the purpose of using it in the first place. One easy story to tell is a situation where your Microsoft Copilot creates a summary of the meeting. That summary gets sent in email form. Then another Copilot summarizes that meeting into a set of bullet points so that an executive doesn’t have to attend. However, the summary creates a lossy compression that mistranslates the intent. Will another meeting get called? Will the exec read the notes? Listen to a recording?

I’m curious what sorts of heuristics are going to pop up for when you should dive further and when you should let it be. This especially strikes me for executives and people who spend most of their time deciding what to dive in based on summaries already. Will “swoop and poop” get worse?

4. What interfaces will pop up that take advantage of the functionality better?

Typing in a question and asking for a result honestly kind of sucks. It’s helpful when you’re spinning up brand-new things like software tests or a new framework or microservice, but copy-pasting code back and forth kind of sucks. And there’ve been a decent number of situations where I’ve found myself troubleshooting the communication as much as, if not more, if I’d just been doing the work myself.

Something like the soon-to-be improved Github Co-pilot seems like a much more natural interface. And I’m curious to see how this will evolve and improve over time.

5. How will customer service work?

I was reaching out to my health insurance the other day and asking about the coverage for a specific set of procedures. I was awed by just how frustrating the situation was, both to have to sit to and to try to understand. It’s easy to imagine why I might want something like ChatGPT to abstract that interaction away from me. I give it a set of commands/concerns and let it handle the communication. Similarly, it’s easy to understand why the healthcare company would want to do the same…

But… that leaves us in a really silly place where we’ve just exposed a much more poorly contracted set of APIs talking to one another. There are many cases where more direct communication would actually be preferable. What happens then?

So many parts hit home, and I’d so so so recommend you just go watch it. But here were a few parts that especially stuck out to me.

https://youtu.be/AKoy_KMVIyU

1. Thought Leadership Doesn’t Exist

There are people, leaders, whose primary value to a company comes in the form of the things they say. The uh… thoughts they have. But it’s not “thought leadership”, thought leadership is someone who wants you to pay them money for an abstracted notion of value without them having to do any actual cultural work.

Nonetheless, the value that gets invoked is the tip of the iceberg of what’s actually happening. The real work is in the social connections they have made, the trust they have built, and their technical understanding of the problem at hand. Tim Schafer, the head of the company, puts this so well when he talks about how two people can say the same thing, but one gets ignored and the other is appreciated. That means your value isn’t just in the ideas you have, it’s in your ability to be heard by the people you’re trying to tell them to. That’s a social thing, not a purely intellectual activity. Yes, the output is “thought”, but the work is pretty nuanced.

2. Double Fine’s Documentary is Probably the Best Piece of Media on Psychological Safety I’ve Ever Seen

The term “psychological safety” gets thrown around so much these days that it seems like a vague description for letting people enjoy their work. Like it’s just something you think about once, make some adjustments, and then you’re good to go. What’s so powerful about the documentary is how it lays bare just how hard it is to develop and how easy it is to accidentally break, and how hard it can be to recover once it’s gone. DoubleFine as a company is an industry standout in terms of caring about their employees, working conditions and trying to do the right thing. And it’s still incredibly clear that they struggle with it, and they still make mistakes.

The line between good and bad leadership is, still in my opinion, so slim. There just aren’t many opportunities to see other teams grappling with it in a real and meaningful way. It was nice to see the reality being represented as this active process that requires constant energy and attention over time, and not this one of thing that you implement, and then you’re good to go forever.

3. Semi-Permeable Boundaries are Everything

One of the most interesting points of tension in the show is overlap and roles and responsibility. Much of the toxic and reasonable tension comes from the overlap in who gets to make “design” decisions about the game.

Game Design is such a great microcosm of this because when you think of “design” you often think of the people making the challenges and building the levels. But as the series makes clear, artists get to make design decisions with the themes they pick, and systems engineers get to make decisions by what they decide is easy and difficult, and writers get to make design decisions. So, it’s all ostensibly a place that everyone plays in, even if everyone doesn’t own it. And collaboration happens when everyone gets to have some say in that. But, collaboration also stops happening when everyone has every say in that. And finding those boundaries is a challenge that changes from project to project.

4. Being a good leader means people will still be vaguely frustrated with you all the time, and you just get to deal with that

In an unsurprising amount of time, employees shared frustrations on ways that Tim Schafer was or wasn’t getting involved enough in the project. What was interesting wasn’t that these occurred, it was mostly that… regardless of how good or bad things were going, the level of frustrations remained fairly consistent. The number of times Tim’s tics get called out, interpreted or diagnosed is pretty often. This isn’t some sort of “gosh, people should be kinder to their leaders”. Hardly. It’s the idea that as a leader, you will frequently get very specific responses that can be informative, but that you can’t interpret as direct causal feedback. The goal isn’t to never frustrate anyone or to be well liked in all situations.

DALL-E Rendering of Being a good leader means people will still be often vaguely frustrated with you and think you could do a better job

It’s an acknowledgement that most leadership is judged moment to moment based on how that person made them feel. And every so often, many times, it will rightfully not be great. And also, sometimes, and many times, that feedback is both true, and not useful because doing the thing that felt less great in one moment leads to a better outcome overall. There’s that quote about the stock market being a voting machine in the short term and a weighing machine in the long term. I think leadership is clearly similar, it’s judged by popularity in the short term, but its true value is determined by trust in the long term.

5. Discomfort is Where the Good Stuff Happens

I’m probably alone on this one, and god knows a terrible meeting is, in fact, one of the circles of hell. But working with other people on tough problems and executing them in a timely manner is, in fact, the best part of it. Half the reason solo projects are such a grind is that there’s no one else to bounce ideas off or to collaborate with. The Human part of technological work, is in fact, the fun part. You can see this in the followup interviews about the group discussions about levels or timelines. So much of the important work actually happens when people are uncomfortable or have to push back against each other or grapple with challenging concepts.

The friction of working with other people is also the reason that working with other people has multiplicative benefits. So leaning into the situations that feel like you might be calling someone out or that you might be nudging too hard, or asking uncomfortable questions. That’s the stuff that moves projects forward.

6. Discomfort is Also Where the Bad Stuff Happens

The counterpoint to this is that capital-b Bad things also happen with what looks like discomfort. When the team goes through what is transparently a very challenging time, it was clearly hard for company leadership to distinguish the types of discomfort. Task-based discomfort, we have a sincere disagreement about the direction of this project and identity-based discomfort, this person isn’t listening to my ideas about the direction of this project can look very similar at the moment.

When we imagine toxic work environments (or really just failure states in general) we think of catastrophes, or situations where it’s obviously just so bad that there are warning signs going off anywhere. But often people are good at normalizing the positives and the negatives. So good stuff can seem mundane (and even uncomfortable) and terrible stuff can also seem mundane (and really uncomfortable). The number of people in the episodes who were basically muddling through but also admitted to feeling super burnt out was incredibly telling.

7. The Whole Leading versus Managing Thing is Kinda BS

The company hosts what it calls “Amnesia Fortnights”. It’s a two-week period where everyone pauses what they’re doing and focuses on small hackathons to develop new projects and ideas.

These episodes are so heartwarming because you get to see new people take the lead on projects and get a chance to exercise their ideas and guide teams through execution. On the one hand, each of these people clearly has exerted some informal influence within the company previously, both because their pitch got voted to the top, and also because they feature somewhat prominently in the docuseries. But the interviews with each of them (and with Tim) about the experience are telling. It’s a new set of challenges, opportunities, and ways of thinking about the world. For some people, it’s an opportunity to take on new responsibilities in the future, and for others it’s a recognition that maybe they’re not as interested in that type of role. But in both cases it’s an opportunity, and an important one, that Double Fine uses to expose leadership to a broader array of people.

The idea that you’d need to expose this sort of ad-hoc “not real” leadership opportunity runs counter to the idea that leadership is management without the BS. If all you had to do was “lean in” more or “lead from the bottom”, specific leadership moments like this, especially at such a caring and thoughtful company, simply wouldn’t be necessary. But, they, in fact, are because often the little tasks, the blocking, and tackling of management, are what make actual breakthroughs possible. And from an equity perspective, leading through influence is a poor substitute for being the one out in front.

Occasionally It's Nice Not Having any Clear Answers

So much of writing about business and leadership traffics in clear, definitive answers about what to do to make things better. You should create emotional safety, listen to your employees/coworkers, and eliminate crunch. But actually doing that in the context of trying to make money, meet industry demands, and also produce good work is a complex challenge. Seeing people take it on, and mostly succeed, is helpful, even if there aren't any clear answers or takeaways.

Since Google’s Project Aristotle there have been numerous articles about how to cultivate psychological safety within teams. But this is all from a top down perspective, as a manager. What if you’re an individual contributor? How do you make sure you’re playing the right role? Especially when it comes to trying to figure out whether to say something and what the right way to say a thing is.

As I talked about in my :reflections about the psychodyssey documentary, discomfort is a significant part of daily life on tech teams. Especially ones that are trying to solve challenging problems. Finding the right level of discomfort is crucial to productivity and happiness. A little bit of discomfort can indicate a healthy amount of task-based conflict, encourages teammates grow, bring up productive disagreements, and work towards an optimal solution. But excessive discomfort can hinder productivity and destroy the psychological safety of the team.

Replacing Discomfort as a Sign-Post

From a personal perspective, I tend to read discomfort as a sign that things are going wrong. It’s bad… right? And I often find myself taking a step back and trying to reinterpret signals of discomfort as something other than bad. Productive discomfort is good, but in order to be ok with that feeling, we have to replace discomfort as a signpost with other things.

Dall-E: a sharpie sketch of a team of people at a whiteboard who are disagreeing productively

Silence

How can you tell if you’re on a team that’s in a healthy space when it comes to psychological safety and task based conflict? I’ve found that the clearest negative indicator is silence. If teams aren’t sharing direct, specific criticism, or voicing(/writing) disagreements, it tends to be an indication that there’s not enough productive discomfort going on.

Finding the sweet spot of discomfort

What’s interesting is these lack of feedback situations can occur for wildly divergent reasons. A team might be socially content and not want to share constructive feedback because it will disrupt a feeling of social harmony. Or, a team might be in an incredibly toxic situation socially/emotionally and not want to speak up for fear of personal or emotional dismissal or retribution. Distinguishing between these types of conflicts can be challenging.

Personal Opinions

On a personal level, what this really comes down to is understanding how and when to share your opinions. There are plenty of reasons I’ve struggled with discomfort when sharing my opinions:

• Strong beliefs that I’m concerned about getting into a large conflict about
• Being unable to articulate an opinion I might have
• Being concerned about inadvertently insulting someone with a critique of their code or their approach

The short answer, is, you should try to push through your discomfort and share your beliefs/questions/in-articulate opinions when necessary. One thing that’s worth remembering is that if a team is in a good place of productive discomfort, it’s ok if you shift a bit in one direction or another (social harmony or too much conflict). Sometimes it can feel like the standard is to not ever say anything that might be inadvertently offensive or hurtful. This also applies to personal discomfort about being told you’re wrong or your perspective was inaccurate or incomplete. I’ve found most people are very forgiving about polite, thoughtful experimentation and mistakes when it comes to

How to Push Through

I know people say you should be more assertive and apologize less. But I’m pretty skeptical that apologizing is ultimately a net negative. If, especially at the start, you need to preface something with sorry in order to feel comfortable pointing something out, you should say sorry. The net outcome of the insight you have to share after the sorry is worth way more than the cost saying nothing might have.

It's important to remember positive/negative team conditions oscillate over time

But, here are a list of framing techniques I’ve used to help push through discomfort I’ve felt when trying to share or open up with a team:

• Be clear about your awareness of your perspective in the critique
  • I think, I see I notice
  • This leads me to conclude
• Be as specific as possible about what you’re trying to articulate
• Stay away from associating any critiques with someone’s identity or beliefs
• If you have a vague feeling but can’t articulate clearly, call that out to start
• If you feel frustrated about a particular situation, call it out. Be honest about your emotional response and specific about what triggered it

What If You’re Too Far One Way or The Other?

Let’s start with the less fun one. If you’re on a toxic team, it’s important to remember that this is one of those situations where, as an IC with leadership responsibilities, you have to decide if you’re committed to being there. And that will be some of what you’re trying to suss out through the process. What I would tend not to recommend doing is bringing up the concerns in a public forum. I would try to approach the issue with a trusted sponsor, outlining specific instances you have concerns about and connect them to the company’s values where you believe things are being missed or let down. It’s also worth noting in this situation it’s very possible you will be labeled the toxic element. And that’s… going to be rough, but will help inform whether or not this is a place you can be.

Especially in this economic environment taking these steps and making such a decision can be a real challenge. If leaving or pushing aren’t options, taking a step back from the leadership role, or removing yourself from the situation could be short term answers.

And now for the happy one. Teams that have too much social contentment are in a better place, because you have more space to nudge a bit. What I’ve found the most successful are finding specific principles you’d like the team to aspire to, and give direct feedback related to those areas. This is where discomfort can be a good guide. You should give (polite) clique that makes you feel uncomfortable, and see how the person responds. And be patient. The goal here isn’t to go from 0 to task based conflict overnight. It’s to help open up areas where people feel like they can give critiques

Experimentation is Key

Ultimately, the only way out is through. Interjecting, raising concerns, and finding ways productively introduce task based conflict isn’t a perfect science. The growth step is to move away from looking at discomfort as a negative, and looking more at the outcomes that discomfort creates. Did it lead towards better project outcomes? Worse project outcomes? Do people feel more open and honest because of direct conversation? Or did something you say inadvertently flatten or discard someone else’s opinion’s? Be on the lookout for these sorts of outcomes as feedback that can guide you in the right direction.

(I'm not receiving any compensation to promote this game, I just think it's a great game.)

It’s Simple Enough to Explain the Rules in a Few Sentences

Everything, for me, seems to follow from this idea. There are only a few components and I could describe them here (in fact, I suspect you have a pretty good sense of the game just from the GIF):

1. There are two types of ships: pirate, and merchant ships
2. Pirates need to get back to their cove and merchants need to deliver their goods and leave
3. The ships will move in a straight line unless otherwise directed
4. There are three types of commands you can give ships, turn right, turn left, and stop.
5. A ship will receive a command when it moves into the space. It will pick up the command given, and if it’s holding one, put the one it’s holding down.
6. The simulation will continue progressing, one step at a time, until success, failure, or getting stuck.

There are other nuances here. But it’s straightforward. Unlikely the two genres I would most closely associate this game with (Sokoban and Automation), the rules are simple enough to explain that you could reasonably set one up in physical form and orchestrate it yourself.

It Feels Similar to Two Popular Genres, But is Wholly Itself

Konkan Coast feels similar to two popular puzzle gaming genres, Sokoban, and automation style games. Sokoban is a style of game where you push boxes around in a constrained space to solve the puzzle or get out of the room. The challenge lies in the fact that there are a restricted number of movements you can make without accidentally blocking yourself in. Sokoban style games often interest me early on, but they get too challenging very quickly, and getting trapped in similar movements because you’ve accidentally blocked yourself can get quite stressful (for me anyway).

Konkan Coast feels similar to Sokoban because the board is finite, and there are only a few moves you can make on any level. You have to figure out what the correct location for each of the tiles is.

However, it also shares some DNA with automation style games. Automation games, like Factorio or SpaceChem or Human Resource Machine, are simplified and abstracted visual coding systems where you put machines together and build them into complex orchestrations to solve the problem. Unlike Sokoban style games, they are expansive and expressive. Since I started programming full-time, automation style games have been less interesting to me.

Konkan Coast feels similar to automation because you are giving discrete commands which you must sequence appropriately to solve the puzzle. When you are done, you get to press play and see how your machine works, from a distance.

DALL-E Rendering of a color pencil sketch of a boat pushing a box on a grid

For me, it contains the best of both worlds. It has the lovely command elements of automation style games, but it has the bounded context of Sokoban style games. It’s great being able to lay down a few tiles, press play and just watch what plays out.

And the components of the game also tend towards the physical as well. You can easily pause the current execution or step forward and backward through it, like a detective scrubbing through the footage trying to catch the right moment to change your understanding of the game.

Story and Game Feel

The story feels competent and well-placed to help keep the game moving forward and give some fun background. It pokes enough fun at the concept (they’re pirates making an automation tool!) without being over the top. And the story fits with the theme, the game is about using a tool to chart the course on the sea most effectively. It’s easy to imagine a bunch of pirates sitting over a table, moving around tiny wooden ships to plan their latest attack.

I did this one, incase you couldn't tell.

This leads to a game feel that’s absolutely off the charts. The game uses the metaphor of a tape recorder to reinforce its tactility. It also makes the process of stepping through the game easy to understand (with rewind and fast forward). On top of that, there’s a wonderful feature where after you beat the level it will automatically rewind itself back to the start to show you what pieces you’ve set. The visual marker of a rewind is a clever way to show that you’re done. And more importantly, the rewind gives you a chance to see what moves you made that actually got you to your success in the first place! Often when I play puzzle games like this, I’ll spend so much time guessing and checking, that winning comes as a surprise. The rewind gives me a chance to see and understand what I did right.

It’s a Fun Game

If you like logic puzzles but don’t want to burn your brain, I would strongly recommend Konkan Coast. The only critique I have, which I would give to almost every game I play, is that this feels like a game that would work really well on an iPad, and I wish I could play it there too.

Check it out!

I decided to try it out in a new project, because it seemed like a simple document datastore would be good enough for the project, and it could be good to learn.

It wasn’t until I started reading The DynamoDB Book and with a gentle nudge from a coworker, that I understood that DynamoDB wasn’t just a JSON data store, but a different way of modeling and seeing the world of data, built on different concepts.

Caveat

My knowledge of this is still super thin. I’m sharing this in the hope that some of my very early learnings will help other people as they learn about DynamoDB.

DALL-E: A visualization of the abstraction of a nosql database as a metaphorical representation of a physical thing in the style of a 14th century oil painting

Things That Changed How I Think

1. Going in I assumed that Dynamo was a simple key store. You could think about it like any other set of files with an arbitrary structure. I was wrong.
2. Dynamo isn’t just a key value store. It can also be what The DynamoDB Book calls a wide column.
   1. The indexes and secondary indexes mean you can create de-normalized entities that allow you to quickly search for data across multiple columns in a whole new way.
3. Data duplication is your friend
   1. As the point that The DynamoDB Book makes, a Gigabyte now costs 3 cents (and getting cheaper) as of 2014.
   2. This means that rather than trying to make sure multiple rows don’t store similar data, the goal is to store data in such a way that makes it as easy as possible to retrieve it quickly, even if it means the same data gets duplicated a ton.
   3. This, however, means developers will have to think differently about their mental model of data storage.
4. It uses HTTP as the connection model instead of TCP
   1. This is part of the reason they work so well with serverless functions. You don’t need to worry about grabbing and releasing connections from individual instances.
5. Your field restrictions don’t have to be quite as intense as they do in Postgres.
   1. For example the concept of doing something like
      1. id: MyEntity#1234 would seem absolutely obscene in a Postgres world because why would you ever need to reference the type within the id. But if you want/need to store multiple similar types within a single instance, Dynamo doesn’t have… well… anything against that.
6. There are differences between Dynamo and other NoSQL type databases
   1. I didn’t fully realize that they had their own flavors.
   2. I assumed that every NoSQL storage was basically the same. But... like everything in development, there are lots of different ways to think about data storage.
   3. Cassandra is open source database structure that behaves very much like Dynamo but without being locked (or supported) in the AWS ecosystem.
   4. Mongo is more document oriented (rather than key value pair) and allows for easier restructuring of data

Things I Hadn’t considered Until I Read the Book

1. The importanc eof Infrastructure as Code compatibility
   1. As the book points out, DynamoDB works incredibly well with infrastructure as code because all of the component parts can be encapsulated in a template or a CDK.
   2. There’s some distinction between this and running migrations. It’s still… IAC, but it lives in a different location and you have to maintain it on your own.
   3. The biggest benefit here is you don't have to worry about things like... say... making sure you're migrating data appropriately as part of a deploy step, or rolling it back if it fails. Your data management can be wrapped up into every infrastructure change you make by default.
2. DynamoDB actually requires more upfront considerations than Relational
   1. I went in with the assumption that in some sense, because you were using it as a key value store, it could change relatively simply over time.
   2. But… in fact it’s the opposite. While DynamoDB is changeable over time, thinking intently about what you care most about right now from a storage and retrieval process, helps make it easier to reduce toil in the long term.

Questions I still Have

1. When, in this new world should you use Postgres instead?
   1. There are probably some answers around when you really need things like “row locking” or immediate consistency
   2. I found the “when to use Postgres” section somewhat uncomplying, because it really made it seem like Dynamo is probably the choice unless you want to get away from AWS lock-in or can’t think about the up front design decisions quickly enough.
2. Why is DynamoDB a faster storage mechanism than Postgres if it connects over HTTP? My mental model says that should be slower.

Setting it up on my local

I wanted to walk through this because getting my hands on stuff helps me start forming better pictures of what’s happening, even if it’s at a super basic level. Things like understand what the bare minimum requirements are to set up a database help give a picture of what constraints are the most important and which ones can change.

As I mention in another article, the purist answer is, of course, to not run things locally. Unless you’re doing something like AWS Amplify. From AWS’ perspective you should be spinning everything up in the cloud. However, DynamoDB is one of the easier ones to relatively quickly have something up and running for your perusal.

With the help of Dynobase documentation, I found the easiest way to spin something up is to run it inside of docker.

docker run -p 8000:8000 amazon/dynamodb-local

This gets a dynamo instance running locally. From there we can use the aws sdk to create and delete instances as necessary.

Either via the command line:

aws dynamodb create-table \
--table-name BookMatcher \
--endpoint-url http://localhost:8000 \
--key-schema \
AttributeName=GameId,KeyType=HASH \
--attribute-definitions \
AttributeName=GameId,AttributeType=S \
--billing-mode PAY_PER_REQUEST

Or in Typescript:

import {
  DynamoDBClient,
  CreateTableCommand,
  DeleteTableCommand,
} from "@aws-sdk/client-dynamodb";
const client = new DynamoDBClient({
  endpoint: "http://localhost:8000",
});

const tableName = "BookMatchGame";

const primaryKey = "GameId";
const sortKey = "Round";

const createTableCommand = new CreateTableCommand({
  TableName: tableName,
  KeySchema: [
    {
      AttributeName: primaryKey,
      KeyType: "HASH",
    },
  ],
  AttributeDefinitions: [
    {
      AttributeName: primaryKey,
      AttributeType: "S",
    },
  ],
  BillingMode: "PAY_PER_REQUEST",
});
const response = await client.send(createTableCommand);

From there it would be possible to write some code that retrieved data, modified indexes, all without racking up AWS charges.

Conclusion

The biggest cost seems to be developer speed, to be honest. (Though, as I’ve grown in my career I’ve become less and less convinced that developer speed as a measurement is all that it’s cracked up to be.) With DynamoDB you can’t just start slamming your hands on the keyboard and cranking out a database. You have to be thoughtful about how you want to store and retrieve data. This, for me, falls in line with the idea that storage options are expanding more broadly, and you should care deeply about how and when your data is being access, instead of thinking about it purely in terms of spinning up some code and some databases.

This is all super basic. I haven’t gotten into most of the more complex concepts of storing or representing data. To the point, it’s definitely a different way of iteratively thinking about a system. In Postgres land modifying data is as easy as spinning up a new table or dropping and creating some new columns with new constraints. But in DynamoDB world, changing things like your primary access patterns aren’t quite as cut and dry. From a development perspective though, that might be as straightforward as getting comfortable with dropping and recreating databases as necessary in order to get up and running, rather than relying on iterative modifications.

This is honestly, quite exciting. Beyond the serverless-ness of it all, it's cool to start playing around with different ways to store and transport data that match the needs of the problems I'm trying to solve.

This is a thought I had the other night as I was messing around with DynamoDB that trying to differentiate (specifically) what the AWS SDK and CDK are for is something of a challenge.

The comparisons between the two seem rather obvious. They’re both packages that wrap api commands in code that make changes to services within the AWS sphere. If I’m being kind to myself I probably intellectually understood the difference but it didn’t really click until I had the thought that I was using the SDK to make direct changes to a Dynamo instance on my local computer, but I absolutely wouldn’t do that with the CDK unless I had spun up some sort of provider like Localstack to emulate the entire AWS space.

Dall-E: An impressionist painting of a cloud and a cd-rom side by side made of tiny strings of code

Direct vs. Indirect Changes

The SDK (Software Development Kit) is a wrapper for API level commands that you have access to in AWS. Making changes via command line or the SDK should roughly have the same results, whether that’s spinning up a new database or triggering a lambda. If you can do it in the CLI you can do it via the SDK.

Where the CDK differs however, is that it doesn’t make any changes directly, and as such, doesn’t have access to do something like… directly invoke a lambda. It’s still possible to do that (if you really wanted), but you wouldn’t really want to do that. What the CDK is good at is creating a chain of orchestration steps to make sure that you can always spin up the AWS services you need on demand.

Idempotency

This leads us to, in my opinion, one of the core difference. The CDK is idempotent. It means that if you tell the CDK to spin up a new DynamoDB instance with the name Fred , you can trigger that command as many times as you want, and it will successfully create Fred once. If you update the CDK with changes to Fred it will modify Fred if possible based on the delta between the existing instance and the new one.

Whereas with the SDK, if you trigger a create database command for DynamoDB, to create a new database called Elouise your first Eloise create command will succeed but every subsequent one will fail because DynamoDB requires you to have unique names within the same region.

Connection to the Stack

The other awesome power of the CDK is that the resources you create are directly connected to the CloudFormation Stack that gets spun up. (This is how it knows how to check for deltas when making modification) That also means that tear down is exceptionally easy. You don’t have to do things like define special or custom names for all of the component parts in order to delete them in the correct order. If you want to get rid of the services, all you have to do is delete the stack, and CloudFormation will (try to) gracefully delete the different components.

Why is There Overlap?

Well, because there might be reasons you want to orchestrate some behaviors via CloudFormation but execute them via code in other cases.

The idea of a DynamoDB instance springs to mind. If you’re creating the instance that will hold all of your data and manage your system for a service you’re creating, using CDK would probably make the most sense because it lets you set the parameters once, and then hook those rules into your CI to make sure wherever you deploy your code, you’ll get the same database.

If, however, instead you want to spin up bespoke DynamoDB instances on demand as the result of some sort of single tenant command as the result of a request or user input, the SDK is more your speed. The SDK gives you the ability to create and destroy the instances you need, as you need them.

In the case of a web application, the questions might be as simple as "What is this user's name?" or as complex as "Tell me the average amount of time a user spends on your website when they're making a purchase between $50 and $100".

The fundamental unit of representation within a database is a key:value pair. A key is the naming structure for some unit of information and the value is the unit that that information represents.

And the key value pairs are grouped into sets of data called rows. You can roughly think of these like Excel spreadsheets. There are many different types of databases that are suited to different types of purposes, but fundamentally they're all groupings of rows with key value pairings of information.

Relational Databases

For a very long time relational storage has been the predominant way we think about and store data within a database. It's so named because the value of the database is in the relationships between information being stored. Relational databases rely on "Tables" which are unique groupings of rows. You can basically think of these like different tabs in an excel spreadsheet. One of the core facets of relational databases is that you shouldn't duplicate data, instead referencing the relationship via a "foreign key". A foreign key is basically the number and the value pointing to the id of another relationship.

The benefit to relational database is that it's always internally consistent. It's impossible to set a value of a relationship that doesn't exist, and it's impossible to remove data in such a way that would make the relationship disappear. This is a powerful modeling technique and a good heuristic with which to think about the world. For example, if you wanted to create a simple model of an author with books, you could create two tables. An authors table with a list of authors, and a books table which links to those authors. This compounding set of relationships can help maintain internal consistency and ask relatively complex questions based on the relationships between unrelated tables, by following the linkage between relationships. I.e. If you were searching for a book, you would be able to access it by looking through genre, author, or edition. Any of these might point you to a reference that you could then use to find the book.

However, this consistency comes with a heavy cost. In order to maintain its internal consistency, before any data is added, updated, or removed the database has to check whether or not it is allowed to do such an operation. Which means if you are trying to do many many of these things at once, the system can grind to a halt.

Document Storage

Document storage to the rescue! If there's been one trend in the last 50 years of computing technology it's that the cost of storing stuff has only gotten cheaper and cheaper. Now it's cheap to buy a small flash drive that stores multiple terabytes of data. This paradigm shaft has had implications for how we think about data storage. Namely, that the whole "not duplicating data" paradigm is silly and impractical. The cost of storing data is trivial, so trivial in fact, that it's smarter to duplicate it rather than to make it internally consistent. Far better, document storage argues, is to focus on the actual expensive part, the act of creating, updating, or removing a record.

If you think about a relational database from a birds eye view seeing the connections between different sets of data, a document store is more like a profile view of a decision tree figuring out how to "access" that data.

Instead of organizing data in tables, or logically consistent forms of key value pairs, data is organized by indexes, or the way that information is typically accessed and retrieved. If we continue our library metaphor, instead of referencing a single book in multiple different ways, you would simply buy multiple copies of the book, and then put it in all of the locations one might expect to find it. If you had a genre section of your library, you would look in the appropriate genre and find a copy there. If you had an author section, you might find it there. There would be multiple books stored in multiple places based on how you expect to find the book.

The downside to this, of course, is that now if something about the book changes you can't simply update one location and be good, you have to update every location. But, in our magical sorting library you could simply ask for all of the books of a single title, and update all of them at once.

The downside to this sort of behavior is that you can't guarantee that at any given time, the information stored will be 100% up to date, because the database isn't making any effort to do so. You can only guarantee that it will be eventually up to date. One of the biggest challenges of this method of data storage is twofold. One, it's a completely different way to think about the world. Even if it's complicated to map out the relationship between books and authors and genres (there are PhDs dedicated to the library sciences), it's a somewhat intuitive way of thinking about the stuff we might want to represent in code. But thinking about data in terms of how you fetch it is foreign, both because it's not how we're taught, but also because it's challenging to visualize the act of accessing data.

Pros and Cons

At the start I talked about the fundamentals of data storage being able to quickly, accurately, and safely retrieve the appropriate information at the right time. And a developer's choice of which location they want to throw their data will depend on what each of those means to them. For example, if accuracy means "I must be able to freeze time at any point and have 100% confidence that everything is good", something like document storage might not be the best fit, because say a book genre might have changed, and a book that should be updated to "romcom" is still being called "romance". However, if quick means, I need to be able to snap my fingers and get an answer immediately, a relational database might not be all that helpful for you because you might have to wait at the end of a really long queue so that the database can tell you that it's very certain about your answer.

DALL-E: A simple sketch of a person in profile view looks off into the distance to see a stack of cubes like 3d excel spreadsheets

There are no free lunches, only tradeoffs you can make and come to terms with as a developer.

I Built a Game

One of my favorite board games of all time is called Decrypto. In it you have a set of four words you're trying to share back and fourth with your teammates (who can also see the words), but you're trying to do so in a way that won't tip off your opponents who cannot see the words, but can use their powers of deduction to figure out what's going on.

One of the reasons it's single player is the simple matter that trying to find people to reliably test a multiplayer online game is challenging. However in the past, the sorts of tools and APIs that were available to make these word associations real weren't all that great. They didn't have the sort of flexibility and randomness that make word association fun. So when ChatGPT came out it felt like a great excuse to play around with a tool that seemed more capable of creating interesting associations that might be able to support a game.

I built a single player variant of this game called Cipher, where the basic goal was a guess and check game where players would be given series of words and asked to categorize them, until they get the opportunity to guess the word. I didn't want to keep the same Cipher concept, but the idea of categorization seemed straightforward enough, so I figured, why not play around with book categories.

The approach I took to this experiment was basically to have an API call to ChatGPT (3.5 I should specify) behind a GET or POST route, making a specific request to ask it either to categorize books or to give an explanation on why it categorized books the way it did. It's not a good game, but it is something that could almost be called a game. The goal was to build a UI where users could drag and drop the categorized books into the slots they felt fit best, and then be explained to by ChatGPT why they were or weren't wrong.

Stuff I Ignored From The Start

There's a world where the UX on the game is nice, where the code is well formatted and where it's easy enough to add on. But ultimately that wasn't my goal with the game. My goal was to understand the functionality of the bot.

I was also interested in understanding what the game might say about the relationship between myself and the algorithm. These sorts of weird game-y expectations outline how we have different experiences about what language is and isn't. And in a lot of ways it did that.

The Game

You can play the game here: https://card-cataloger.netlify.app/. (It is very rough and prone to erroring)

The game occurs over two rounds.

At the beginning of the game, I fetch a random date in the last 10 years, and get 14 randomized books from all of the New York Times best-seller lists on that day. I then pass those books onto the ChatGPT prompt asking it to categorize them into 3 categories.

Dragging and Dropping Books

The 14 books are handed off to the frontend which then splits it into two hands of 7 (or maybe less if I didn't get 14 books back). Players are asked to sort each of the books into the category they think fits best.

Seeing A Description of the Book

When the round ends they're given an explanation of why they were incorrect.

Showing the Results After the First Round

Then they get the second hand of books and have to categorize the rest, with double points being awarded for correct answers. And that's it!

Final Results

There are definitely a bunch more directions the game could go, but if you want to mess with this Proof of Concept, you can find it here. It will probably be up for a bit longer.

The Boring Stuff

The game was built on top of Parcel, React, and XState for game management on the frontend and Chalice deploying to an AWS Lambda on the backend. It uses OpenAI's ChatGPT turbo and New York Times' Book API to source data.

I suspect there will be two primary critiques my conclusions based on this approach:

1. I'm using 3.5 and ChatGPT 3.5 is significantly less good at what I'm asking it to do than ChatGPT 4 is.
   1. Yes that's true, but I'm not so sure how much it matters in this case. The takeaways will likely be the same (if somewhat less oftenly occuring)
2. I'm not using ChatGPT the "right way".
   1. As many tutorials will explain, the correct way to "use" ChatGPT is to throw it into some sort of chat based interface. I'm just still completely skeptical that this is the way most of these things will function. I think prompts will ultimately become wholly overrated as models get "smarter" and as the best prompts are agreed on and able to be re-used effectively. The most exciting use of AI I've seen thus far still comes from Copilot where it acts as a sort of superpowered autocomplete, without requiring me to jump out of my flow to actually function.
   2. Setting a UI in front of this might be unnatural, but in the flow of the game "categorizing books", I don't think asking people to type out a chat based prompt particularly makes much sense, and much of the benefit of the chat interface is that it means that people get to do the heavy lifting of figuring out and ignoring the hallucinations.

What I Realized

Formatting Responses

Even giving very specific commands, the formatting of the response would come back in a variety of ways, with different characters, increments (numerical versus alphabetical), and different parentheses. Sometimes it would come back as [Book A: Title] or sometimes it would be Book A: TITLE or sometimes it would be Book A: [Title] or Book 1: Title or any combination there in. That meant that getting the request back in a format that I could easily decode and use was always a challenge. I know it's theoretically possible to request a JSON response, but parsing JSON from a source seems less than ideal.

The Occasional Hallucination

While I expected hallucinations going in, I had hoped they would be more... well interesting on average than they ended up being. Most of the hallucinations I noticed happened when asking it to describe why a particularly book had been categorized a certain way. The facts of the book seemed to get all jumbled up. I was hoping that the hallucinations would lean more towards suggesting an interesting category or the like, but I never quite got there.

This all goes back to prompt tuning. There's probably a world where I could do more processing to categorize the books, and then a separate prompt using the books to ask for an interesting category. But, it also doesn't seem like the type of thing that you can create clever prompts to get better results. While it seems you can give better prompts to get better answers, you can't necessarily write cleverer prompts to get cleverer answers. (More exploration is probably warranted there.)

Getting Back What I put In

One constant struggle was trying to make sure that the books I put into the algorithm came back out. I would say about half the time I would get a different number of books in total that were categorized, than the number I asked for details on. Even though I almost always got the correct number (and format) of categories. I would also occasionally get a response like "I'm a Large Language Model and do not have opinions", which, given the amount of time I put in, would break the system and require a reset.

For example I had the temperate set fairly high (1.0 on a 0 to 2.0 scale), I was equally likely to have that result in poor formatting as it was to be more interesting or dynamic responses. There are probably other sequences of requests I could make to get the appropriate combination of sequencing and interesting names, but I'm not sure if that sort of time investment is necessarily worth it.

What's the Role Determinism Has to Play in the Future?

When it comes to Large Language Models, the whole nature of the "stochastic parrot" is both the excitement and the drawback to the technology. Obviously whole methods of working will change, but spending your time trying to clean up after that sort of output is neither productive nor particularly useful. It feels like there are two potential paths in the road. We either "get over it" and build out more chat based systems that let human brains parse the subtle differences that might get spit out, or more time and effort are put into improving the structure and determinism of the output.

It's not all that hard to imagine a world where you're inputting all of the drag and drop functionality into ChatGPT instead of creating a UI on top of it, but in that world is the lack of determinism of the experience a drawback? Or part of our lives? Playing sudoku that way wouldn't work the same, for one example. But maybe that's just the cost of further integrating the power of the technology into our lives.

The Game Stuff

One of the most interesting things about experimenting with the game was realizing that actually the input of the books played a huge role in how fun or exciting the output might be. If the books were too closely clustered together in concept, it wouldn't be all that interesting to try to pick them apart, because all of the answers would feel like cheats. If the books were too far apart, the connections inside a category felt more like a laundry list than an insight.

On top of that, I realized that while there were real moments of sparks, when a connection between some disparate ideas would come alive, those were too few and far between. My favorite was still The Hungry Caterpillar and The Mueller Report being both categorized as "growth and discovery", but for the most part none of the connections were revelatory or particularly interesting to play with. The "bad input bad output" comes into play here. Zach Gage talks a lot about how when he started creating Good Sudoku he thought that just setting up the engine was most important, but what he learned was that type setting the challenge was equally important, because most of the possible (valid) puzzles actually weren't all that fun. Doing something similar here feels equally true.

This could come about in various ways. Either doing something like a "Wordle" with daily challenges that everyone has to take, or something more like a versus challenge, where you would get to the pick the books, and challenge someone else (or yourself) to figure out how they might connect.

Game Changes to Make

A "Challenge" Button and GPT

One interesting feature I thought about was the ability to "Challenge" the results. If you didn't like something you could ask the algorithm to reconsider and re-adjudicate the decision. I do actually think this would be an interesting area for the game, but it would only be interesting if the game itself had its own persistent model (or maybe the player gets their own model), where the successful and unsuccessful challenges would create a tapestry of experiences that could inform future community or individual play. It's the difference between training and managing your own model versus using a more standard out of the box formula.

The Word Association Was Quite Good But Not Quite There

Honestly the connections between themes were often quite fun and insightful, and I appreciated how it was able to come up with categorizations that created some sort of interesting play. There are probably better prompts I could've managed (or maybe if I get that GPT4 API Key) to see if the output would be a better structure. But what I realized was that the magic of both Decrypto and Codenames doesn't just come from the rules, it comes from the adversary nature of the other players. Having to create a perfect prompt to try to spit out interesting combinations didn't feel like a productive use of time, and the connections between the books, for the most part, wasn't particularly illuminating or exciting. Most of the feedback I got was "I realized the bot was right and I was wrong", which is interesting, but not all that fun. Fun would be answers more like "That was a connection between two concepts I hadn't considered before".

Not a "This is Useless" Critique

It's easy to conclude that this is some sort of "not ready for primetime" critique of using ChatGPT. But it's not. What I was specifically interested in was the ability to use ChatGPT as a sort of "human proxy" for multi-player games (especially word games), and then as its own sort of language based experience. What I'm interesting in is how sprinkling these sort of language based experiences requires a lot of shifting on the part of how user experience flows work. Either the entire UX has to change to be a chat based experience (which I'm skeptical is the long term goal), or a lot of work has to go on behind the scenes to make sure the inputs and outputs are useful for the specific experience you're trying to give to your user, and having it be a value add beyond something you could do by slamming some prompts into the text box.

The sorts of experience that seem the most exciting to me, are the ones like Github Copilot where the completions are extensions of the thought process you already have, rather than as a wholesale shift to a prompt-based methodology. But there's a lot of tension there, because the prompt format is obviously the past of least resistance, structuring the outputs into a UX format that's not a conversation isn't trivial, which means it can't simply be pasted into applications quickly and easily.

My concerns about the technology are less about the automation of it all (though there's plenty of that) and more about looking at how easy it is to create really bland, middle of the road content, pushing more risk and responsibility on our users instead of using the functionality to give them more incredible experiences.

The idea behind social force is the concept that you can change your tone, sentence structure, and body language to put more weight behind an important thing you would like to see happen.

It's helpful to use in situations where something needs to get done efficiently, and you want to make sure its importance is marked and communicated to others.

One of the biggest areas of struggles I had when I tried my had at management was the understanding of how I could get people to do things. I don't mean this in a negative sense where my reports weren't doing their work. I was thinking more in the literal sense of, at some level there's nothing stopping someone from blowing me off. I was an inexperienced and nervous manager and this concept seemed all too possible. For the most part, in modern engineering teams, personal ownership is encouraged. Leaders are responsible for framing the work and then allowing engineers to take that work and run with it how they see fit. This is really helpful for the majority of situations and the implementation drop-off/disconnects are either non-existent or positive because the people owning the work get to be the ones making the decision. When we talk about leading through influence we're often talking about persuasion, i.e. using the language of someone else's priorities in order to convince them that the thing you want to do is a good thing to do.

Often, as I discovered, as a manager, people will simply do the things you ask. And, the higher up you go, the more likely that your status will give you a good amount of leeway in terms of how quickly people will do the thing they think you're asking based on the stories they've built in their head about you, and that becomes its own sort of problem.

However, there are situations, even in high-autonomy teams, where nudging is important. People need to be aware that their interpretation of a situation isn't tenable in the current context. When you're working with people who are disconnected from you, who are similar level, or if you are trying to push through an issue that you think might be dragging on (like a discussion that's getting bikeshedded), there are social cues that I've seen the best managers and leaders tap into in order to get things done: I've started thinking about it as "Social Force".

It's the Jedi equivalent of a force push, but you're doing it with words instead of with fantastical powers. The basic idea here is that there's a shift in tone and demeanor you'll use to communicate that you've exited the "getting people excited about this potential" phase and entered the, "this is happening and let's figure out how to do it".

Why is it important

Firstly, even good teams will swirl on topics that are ultimately either not up for debate, or not so important as being worth the amount of debate being given. Managers have a duty to step in and stop bike shedding at the source, and social force is an important use of that. Secondly, when working across boundaries, especially at larger orgs, social force is a way of communicating importance about a certain task. And finally, social force is a good way of making your boundaries pretty transparent. One of the challenges of working in a emotional safe environment is that it can make every request feel value neutral, collaborative, and like a group decision. But not everything is. Some things are in fact quite important, and it's important to be able to communicate that without being mean, rude, or unclear.

A color pencil sketch of someone force pushing another person's thoughts

What are the limits

That all being said, social force has its limits. If you go around trying to force push everything in sight the power is going to diminish. Social force is best used when there are clear implications of consequences for failure to go along. What's important here is that the consequence is not firing or dismissal, it's something more uncomfortable, like, "I'm going to annoy you until this happens" or "you won't get to do the thing you want until we get this done". It's not "Do this or your fired". That's the equivalent of a force choke and is a basic sign of a toxic workplace.

This means that social force has similar limits as other sorts of social negotiated behavior. You can only do it so long as other people respect the authority (or the nagging) of the pressure you're putting, and so long as you're not using it so much that it becomes ineffectual, annoying, or destructive to the emotional health of the environment.

How to Do it

There are a few key indicators about using social force that are verbal, even on zoom calls. Speech tends to become more direct, and the response might cut someone off to indicate its importance. There's a change in language from questioning or uncertainty to concrete and action oriented. The statement tends to be a smidge louder, a bit sharper, and end almost terse. Your statement won't go up an octave at the end, it will end firmly. The goal is to create the implication that "this is happening and let's talk about how" rather than "What do you think about this?".

Use It Wisely

In terms of being a good leader, it's most effectively deployed not when it's about "we should do it this way because I'm right" it's deployed in situations where you need to make other people start caring about a situation, project, or outcome that they might not because it's important for the people you lead. If you're not deploying this as an avatar for others, you're probably just being an a**hole. It's not about getting people to do what you want, it's about situations where the social cost of the discussions are higher than the cost of nudging the solution through.

As I said, this isn't the type of thing you want to use often. It's most effective when it's used sparingly, there's a clear outcome you have in mind, it won't destroy trust, and you have a reasonable chance of it working. If you try to nudge people directly in every meeting you're in, you'll get tuned out, or asked to leave the meeting. And occasionally, you will have to back it up. Especially when working across divisions/responsibilities/etc. the "consequences" piece is less about formal threats and honestly more about being annoying. So if it doesn't get the outcome you need, you should be ready to nag and push (and basically create some discomfort) until it gets there.

Use Your Powers Wisely

Like I mention in How to Lead a Good Meeting, keeping control of a conversation as a leader is an important part of creating a healthy working environment. "Social Force" is a tool in that toolbox to ensure productive collaboration. However, it's equally important to understand when conversations shouldn't be shut off, to understand the real costs of using "social force" and learning from the feedback you get about situations when you could have used it better, or not used it at all.

Especially in cross-functional meetings, one phrase I hear a lot is the concept “I don’t think we accomplished what the meeting wanted necessarily, but it was clearly valuable for other reasons”. Or, "it was nice to connect with people". This has led me to the conclusion that one of the primary challenges we have with meetings is the way that humans are so good at finding reasons a meeting we were a part of was valuable. It’s a natural human thing. But unlike other pursuits, it makes it much harder to diagnose when a meeting has gone off track. I think often when it comes to meetings, typically because we’re all very busy people, we become very very undisciplined about the purpose and output of meetings.

One of, and I mean this sincerely, the most impactful pieces of writing I've read in my life is Shit Crayons by Ian Bogost. It’s about games but I think it broadly applies to meeting culture too. Summarized really shortly it's basically saying your ability to derive value from a meeting (or insert your preferred form of cultural engagement) has nothing to do with that thing's value.

This seems counterintuitive at first because the purpose of a meeting is for its attendees to find value, right? Well... yes and no. I'd like to posit a few things:

1. The majority of people who attend (especially high level) meetings are smart thoughtful people who got where they got by finding ways to get value out of situations that are less than valuable (or even downright challenging).
2. People, in our beautiful individuality, like being together, and will generally try to find ways to find meaning in the things we do.
3. A meeting having some value doesn't mean it had the value we intended for it.
4. And especially for point number 3, we conflate them, because of points one and two.

Most of the time, frankly, we don't hold the artifact of the meeting accountable to the same standards of practice that we do our other (more tangible?) outputs. And that's the problem.

I unfortunately don't remember exactly who said it (I think it was Jared Spool) but he mentioned when presenting to executives about the benefits of design he would find them nodding along and agreeing with what he said, thinking it was rather straightforward (and obvious). This was unhelpful because it made it less likely for them to make the changes necessary, and he knew going in they had doubts and concerns. So, he started having them write out their design assumptions before giving his talk, so he could explicitly track and have them compare how their opinions changed.

This is the same problem we have with meetings. Because we don't do a good job of tracking the goal of a meeting, we'll find some value in having gone, and then backfill that as the reason for the meeting in the first place. This is backward and makes it hard to actually get a picture of if we're leading effective meetings in the first place.

Dall-E rendering of a woman at a table in a meeting painting an award winning piece of art when she's supposed to be listening to the presenter

We can be more critical about meetings

One of the challenges we have with meetings is that, because they’re social, we recognize they have value outside their stated purpose, so we’re reticent to put on additional structure that might destroy benefits unintentionally. The analogy I'd like to put forth is one about code. I think there's a lot of value to writing bad code and then learning from it because even bad code that gets tossed is still an opportunity for developers (especially junior ones) to learn and grow and experiment with new ideas. It is possible and desirable to find tangential value in the things we do even if they don't get completed.

But the caveat here is that the code still gets thrown out (or changed or what have you) for not meetings its stated objectives. We don't say, well we learned a lot, so that code must have been good, let's ship it to production. We say, we learned a lot, got some side benefits, now let's go fix it and do it the right way.

We can do the same thing with meetings. We can set goals, aim at those goals, and measure the impact of those goals. We can be forward looking in learning about how to construct better meetings while also appreciating and maintaining the benefits of helpful (but unintended) connections that get made.

How To Fix The "Unrelatedly Useful" Problem

It's one thing to talk about how common this is, it's wholly another to propose actual solutions.

1. Axe them

The challenge to any fix is that they require way more time and effort. That’s always the answer with meetings. You need to put in the prep work and set aside the time to make them valuable. And for the most part that time doesn’t exist. For most meetings that’s going to mean killing a less than efficient meeting that might have some external value. I think in corporate culture it’s really hard to let things drop rather than create a bad meeting, but ultimately that’s the decision we’re going to have to encourage more if we actually want this to go better.

One of the primary reasons meetings stick around is because companies believe that the cost to accidentally duplicating worse is much higher than the cost to sit down and meet. This is an area I would love to know if there’s research on and what the conclusions have been. But my gut says we tend to default to too conservative rather than too duplicative.

2. Create Goals

I talk through this in the previous post, but it doesn’t change the fact that your meetings should have goals. And those goals should not be the content of the meeting. In order to create momentum a meeting should end with your team or you division or your company feeling further along than when it started.

In my previous post, I talk about setting clear goals at the start. But in terms of evaluation of larger, cross-functional meetings I would go a step further. Set specific metrics you intend to measure at the outcome of the meeting. When you work with people on a daily basis it’s pretty easy to tell whether or not what you’re doing is successful.

Goals look like people leaving your meeting with clear action items, meetings creating deeper understanding, or meetings eliminating unnecessary work.

3. "I will know I’ve met my goals when…"

Now that you’ve set goals, you should figure out how you’ll know if you’ve met those goals. At the company level, surveys can be a way of accomplishing this. You don’t need to survey (or talk to) everyone, and you also don’t need these goals to be measurable, but you should have some very clear sense of how you know if you’ve met the goals of your meeting. For example, if your goal for a meeting is to kill a project, you should define how you know that project is killed. If your goal is to create a shared understanding about an important concept, your goal shouldn’t be talked about the concept it should be something more granular and refined, like certain keywords, phrases, or framing you should be looking for.

4. "I know the meeting failed if…"

This might seem like a breathtakingly obvious corollary to the previous point, but it’s important to remember that like people, meetings are multi-faceted. It is possible for one person to have a successful meeting and another to fail, and it’s also possible for both success and failure metrics to have been hit, which might suggest you went in with less than ideal assumptions.

If we go back to our previous examples (killing a project and building shared understanding), you might be on the lookout for something like vocabulary to suggest that work is still progressing or the priorities that made that work necessary are still present. And if you’re looking at shared understanding you might be on the lookout for confusion or soft yesses, or people who are agreeing to understanding but unable to describe the concept on their own.

Tangential Outcomes are Great

I’m worried by reading this you’ll hear that unexpected problems are bad. I’m absolutely not. If you get bonus conclusions out of a meeting that’s awesome. Or if you decide that a meeting should be unstructured time to allow for connections that’s also great. What you shouldn’t be comfortable with is the idea that you brought people together for objective A and you accomplished objective B instead that that’s a positive outcome, especially for a large or recurring meeting. It’s not. It means you have a lot of smart thoughtful people who are good at making the best of a challenging situation.

Part of the benefit of meeting with other people is to build shared understanding. It’s helpful to go into an unexpected path and collect an idea of what other ways others within your team or company have thought about a subject or objective. Meetings can and should have enough space to allow for occasional diversions. But, we shouldn’t let that distract us from our primary goals.

Rip The Band-aid Off

One of the deep ironies of our current relationship to meeting culture is that we despise them so much we refuse to put in the effort to make them valuable. This was one of the most interesting component’s of Cal Newport’s book A World Without Email is about emails. It's really about how information flows at a company. And meetings are part of that. In order to reduce the number of overall meetings, we have to find ways to be more disciplined and demand more out of the meetings we create. And that starts with being more honest about if a meeting was actually successful or not.

Invariably when working (or really interacting) with other people you're going to run into situations where the only way to resolve a misunderstanding is by talking through it with another person.What this typically looks like is feeling the need to communicate a thing, but knowing that the process of that communication has emotional or interpersonal costs. These costs can come in the form of frustration or confusion.

In the radical candor verbiage often this comes in the form of ruinous empathy. We withhold the thing we might want to say for fear of hurting someone or causing confusion. We typically do this because we like to imagine a world where if only we knew the correct or the best thing to say we would say it. But the situations where you can both communicate perfectly to resolve a situation and one’s where the situation happened in the first place are few and far between.

We often imagine this in the context of withholding feedback from someone who we have constructive (or downright negative) feedback to deliver. And that is one of them, but I've seen this come up more often in other situations. I've typically seen this happen when

• you know you don't understand something but you don't actually know what you don't understand
• you see people agreeing with each other but you can't tell specifically what is being agreed upon
• you see people agreeing but you think it's more out of a desire not to actually continue the conversation
• people seem to be disagreeing but also talking about separate things

In all of these situations it's easy to imagine a clever, suave, well-spoken individual stepping in and calmly resolving the situation. Like those scenes in movies about high schoolers where the popular kids walk past and the wind is gently blowing their hair and they look cool as hell. But those people don't exist.

What is "Bad" Communication

DALL-E: Claymation of someone in a meeting saying something other people cannot understand

Before we get too much further I'd like to walk through a non-exhaustive list of forms of "bad" communication, that can still be useful:

• Misusing a term or phrase that's key to a discussion
• Asking a question that seems like it has an obvious answer
• Stating your perspective of the world/problem/widget and being wrong
• Sharing a judgment that is critical and learning it was based on a faulty premise
• Clarifying a point that someone of your status/level should "obviously" understanding
• Critiquing someone else's work
• Disagreeing with someone else's opinion

We assume the path from confusion to understanding is linear and up and to the right. We start a meeting not on the same page, and then we end the meeting on the same page. It isn't, and I'd like to walk through the reasons why.

Communication is a Multi-player sport

It's weird because we do it so much that we tend to assume that communication is easy. It's like water for a fish. This makes it incredibly easy to over-internalize communication that's going poorly. If you don't understand something, it's a failing in your ability to understand, if people don't understand you, it's a failure of your ability to communicate. But the reality is that the state of positive or negative communication is more a reflection of the relationship between the people at the table, rather than any given person. It's why reading academic papers (even about straightforward stuff) feels like reading a foreign language. The relationship between the people communicating is unstable.

There are plenty of ways to resolve this instability. It might require your going and doing research, reading up on vocabulary, or leveling up within your chosen field. But it also might mean that there's a fundamental misunderstanding in what's getting talked about and deserves asking a question or clarification to resolve. But the first step here is to push your observation about how the communication is going outside of yourself by remembering the state of communication reflects the relationship and the moment.

The Stakes for "Bad" Communication are Immediate; The Stakes for No Communication are Long Term

When you say something confusing or incorrect, the looks of confusion are immediate and you'll have to deal with it. If you don't say something confusing and let it go unsaid, you have the opportunity to kick the can down the road, and maybe never deal with it. The costs to disorganization on the project will be distributed or not felt among multiple groups of people. It will just be a vague dissatisfaction. It's less visceral. So we tend to default to the thing with lower immediate stakes. It's like the unholy child of present bias and the tragedy of commons in a TED Talk.

But the rewards for overcoming our fears of bad communication can also be immediate. The most satisfying part of a meeting is leaving feeling like you built a shared understanding where previously there was none. Or a meeting where you left with action items and a visceral understanding of how they connected with one another. "Bad" communication can help build understanding in confusing situations.

We Confuse Poor and Hurtful Communication

Bad communication we think of not knowing how to properly phrase our thoughts or being unable to describe what we see. So we don’t think of difficult communication as bad, we think of it as rude. Because we assume it's easy we tend to create all of these other assumptions like bad or rude communication is always a choice to be mean or rude. If communication is going poorly, and you can say something that will create clarity at the risk of being rude or hurtful or sounding stupid. You need to say the thing that is potentially rude or hurtful or stupid. There simply is not a world where you can get to a better place without actually overcoming the thing that you need to say.

This is not an excuse to be an asshole or to treat other people poorly. It's a recognition that part of learning how to communicate well, both in the short and long term, is by communicating poorly, and using that understanding to communicate better in the future.

The State of a Conversation is Not Fixed

This is where I really want to land. It's easy to assume that as you speak you will either increase clarity and improve conversation, or decrease clarity and make conversation worse. But conversations don't actually operate on those axes, always either getting worse or getting better. Sometimes you have to complicate a conversation in order to get to a better outcome.

This is the same as the adage “anything worth doing well is worth doing poorly”. The solution for misunderstandings, confusion, and lack of clarity is more communication where people are honestly trying to represent their opinions and beliefs, not ones where people aren't saying anything at all.

One of the interesting things about being a software engineer right now is the sheer amount of stuff going on in the world that you could theoretically pay attention to. Even if you're not staying up to date about AI there are still advancements in Frontend development and Serverless development that you could spend all of your time trying to keep up with. It is frankly stressful and can make just "doing work" seem less important than trying to keep up to date.

I was thinking about writing something about picking your battles when it comes to learning new information until I saw this article by Casey Newton about the plethora of note taking apps. I found the reframing from "choosing what to learn" to "thinking about how to build knowledge" helpful. Why note-taking apps don’t make us smarter

It put a bunch of ideas together for me about building and maintaining knowledge in the 21st century. Basically the idea that we have all these tools to build knowledge but it actually feels like things are getting worse, not better.

And I think basically that's because the way we build metaphors about saving information is actually out of sync with how we use it. And we typically run into two problems:

1. Organizing Past Information that we've saved
2. Staying up to date with the information being shared with us.

How We Think About Information Is Not How We Use It

We think of information like streams coming in and libraries in storage. So we spend a lot of time categorizing, bucketing and shifting records around in an effort to keep them close to our mind lest we lose something important. And when we're confronted with a new piece of information the choice is between drinking from the well of knowledge or not. And in that frame the answer seems obvious.

And that's because while we tend to behave like information is a discrete unit of facts that we can collect, organize and expound on, information is actually an active process.

But this is a critical misunderstanding of what "knowledge work" actually is. One of the interesting things about learning about spaced repetition is the idea that how we actually remember a thing has less to do with our ability to understand it and more to do with our ability to be tested on it. Which gets at something key about knowledge work. Knowledge work is about practice, not information.

Competence, when it comes to information and knowledge work is actually about the art of deeply engraining ideas within our neurons such that they can be reliably retrieved under a variety of contexts and circumstances. While we might act like our brains are vast libraries of informations, I would categorize them more like swirling maelstroms of information between which a cursor (our identities) ping bong between concepts. This means the value of merely reading something can be shockingly low.

The Line by John Curry, 1935

Our Software Tools Support These (bad) Metaphors

The interesting articulation in this article is the way that systemization of our personal knowledge stores (and I would add our todo lists) has done little in the way of actually helping us manage information more effectively.

Basically we creates backlogs of information as though they can act as "second brains". And the idea here is that (like Getting Things Done espouses) if we get externalize the actions we need to remember we can simply act on them later. But there are two challenges with this:

1. As Newton notes in the article, the amount of time we have to spend building and maintaining these systems of knowledge is often much more of an investment than their payoff represents.
2. The half-life of tasks, ideas, backlogs we capture is preciously small. To the point where, after a certain amount of time, an old task is as much a piece of "new" information as it is a remembrance of the past.

This means between bucketing and categorizing new information and maintaining old information, it's so easy to spend more time carrying forward past commitments to ourself than it is actually acting on the information that's in front of us.

It's Basically The Same With Incoming Information Sources

Because our society puts such a high value on "being informed", being up to date on the latest terms, vocabulary and ideas floating around can feel as important as executing the work. Not because the words flying around have an immediate impact on our jobs but because our ability to recognize and categorize them correctly feels tied to our social standing. If we don't know them we might feel lost, uncertain, foolish, or the dreaded word "behind".

If we let go of being up to date, there's a reasonable fear that our social standing will change. The value of our knowledge is deeply tied to the social standing we have in our community after all. (Check out Being Wrong for a deep dive on why and how that's the case) It's why disinformation spreads so fast, because if we have to choose between our status and our beliefs, we will tend to pick our status.

The irony of this situation is that "catch up costs" for information have never been lower. It is quite easy to find a few information dense articles and get a general understanding of a specific topic. But what's much harder is building the compounding sets of knowledge that differentiate your ability to act effectively.

Ignorance Is Bliss

The conclusion here to me is straightforward, if maybe painful to hear. It’s beneficial to choose to be ignorant. Being behind on some things gives you the ability to practice and act on information in ways that compounds your understanding and ties it to other knowledge you already have, so you can speak and act with fluency. It's the difference between understanding what a switch case is in Javascript and understanding when it's idiomatic to use one.

Cal Newport's concept that humans at value to knowledge through focus is key here. Knowledge work is active work. It's about choosing to recommit to a single topic repeatedly, which means letting some information literally slide past you. To accept the fact that you will become (and more importantly feel) less knowledgeable about other things because you believe the value of knowledge of the thing that you're focusing on is actually more important.

I’m saying this to emphasize that knowledge work requires active practice. It can feel like knowledge work is just about finding two sets of facts (bundles of words) and drawing a line between them, but it’s actually more nuanced. Part of the reason that executives can sound so weird is that they’re literally out of practice in applying concepts so they can sound like a tourist confidently mispronouncing words based on their extensive Duolingo practice. (I should note this isn’t all executives but when you think of the ridiculous executive, I think this is a large reason why.)

When we think about getting rid of our backlogs (tasks, books, movies) we imagine that what we're doing is being more forgetful, present driven, and ruining our memories. But... I think it's actually the opposite. Instead of letting go of half baked thoughts and ideas we carry them into the future with us, indefinitely, like the ghosts of what could have been, crowding out the space that we could devote to our present.

Where AI Can Step In

It's possible as Casey Newton notes, that AI could be an excellent solution here. Wouldn't it be great if we could simply ask ChatGPT which of the articles in our personal repository of stuff that we thought might be useful one day actually aligns with a present problem. Instead of asking it to hallucinate facts for us, ask it to recall information that we have found valuable at one point, but have chosen to forget when it is relevant to us again.

Leaders Are Important

That's why I think it's especially important for leaders to show good behavior here. This isn't to say that you should ignore new trends, just that, it's ok to delegate the research of bleeding edge stuff to other people you trust. Some people will want to commit to being up to date and actually experimenting with [insert new thing] and signaling to others that they can get up to speed when the time comes.

Takeaways

#1 Personally and professionally try to eliminate, hide, or ignore your backlogs. There are certain elements of our past selves that we want to carry into the future, but it’s worth being intentional and recognizing that

#2 Ignore new information more proactively. Prune your inputs as well as your backlogs. Lean into your social networks and let the people who are interested in new ideas help support your learning goals. If you’re in “wait and see” mode on AI that’s actually OK. You probably have friends who are digging in. Trust them and let them guide you. You can apply these sorts of practices to any number of skills. Invest deeply in the skills you care about instead of trying to stay up to date about what the internet thinks is most important.

#3 There aren't any easy outs. Choosing to not pay attention to some things still means consistently pruning and considering our sources. But hopefully making the decision to eliminate sources is easier and quicker than trying to organize them for future use.

About a year ago I wrote about becoming a staff engineer. I want to evaluate how well my initial thoughts hold up and put some new thoughts out about the role and how it's different from other Individual Contributor roles.

Me, attempting to embody "Staff Engineering"

Art by Henry Dawkins, active 1753 - 1786

Evaluating My Assumptions Coming In

:Code and Culture

My idea here was that staff engineers ship code that change culture. I'm going to call this one a whiff. Yeah... it's kinda true, but ultimately this doesn't happen all that much in practice. And realistically everyone at every level should be encouraging a culture of automation that makes good practices a default.

:Getting Here and Going There

This one about urgency continues to ring true, with the added twist that changing the time scale of the impact I'm trying to have is important. The value of decisions has to be bounded by the start and end of when they will be valuable. There is a half life on decision making that means if I don't take action quickly enough, there's no that much value to making a decision at all. And other things that might not have an impact for a pretty long time.

:My Theory about Wealthy People

I still feel pretty confident about this one. The higher up you go the easier it is to get disconnected from the people doing the work. You have to opt into challenging your perception of the world and staying aligned with the people owning the work.

:First "Bullshit" Role

This is a weird one? My hypothesis that it would theoretically be possible to coast is probably false. Good managers and directors will know whether or not I'm doing good work. At the same time, one of the most rewarding and weirdest experiences I've ever had was when someone gave me the feedback that my attitude towards problem solving made them feel like it was possible to solve any problem. And the level of trust and confidence me being around gives people still makes me pretty uncomfortable. It's weird being in a role where much of the value I have is in the ability for my process to be copied by other people.

I was re-watching The Return of the King and there's the moment where Merry turns to Gandalf and goes "we've got the White Wizard, that's got to count for something" and you can just see the "well fuuuuu**" cross through his eyes, and boy do I feel that. The number of times people will be like "Oh Stuart can figure that out" and I'm going "Oh Stuart has no idea what any of this means" internally. Yeah that's a real thing. But... that is actually also the fun of the job. Sometimes you get to scare off a bunch of Nazgul.

:Everyone has a different vision for staff engineering

This one still rings quite true. Depending on company size, goals, and opinions about titles, both what a staff does at one company might differ and what the expectations for experience vary wildly. I have friends who are much smarter and more experienced who are not Staff Engineers.

Core Behaviors

But what about the stuff I used to do as a Senior Engineer? How has that changed in my new role?

Writing Code

I still write tons of code, but the types of code I write has changed a fair amount. Instead of focusing on building on the core functionality of a project, I spend more time digging around the extremities. I'll work on issues that have stumped other developers, patterns that can be re-used, or maintenance style tasks that involve getting up to standard with monitoring or other infrastructure type work. I also spend a fair more amount of time debugging issues as they pop up or working on tasks that are gnarly to solve but might be higher leverage. I also spend a bunch of time pairing with other devs, walking through implementations and architecture.

The one caveat I'll add here is that this experience was fairly different when I worked with the team I had been a senior engineer on versus the one I moved to already as a Staff engineer. When I remained on the same team as a Staff engineer, I spent more time on specific project execution. Though my experience was that caused way more problems than it solved with my efficiency and institutional knowledge.

Not being directly connected to the core work for a key project can be challenging at times, because emotionally the outcomes of the work is more diffuse.

Writing Non-Code Stuff

Tickets

I write way more tickets than I used to. Instead of building out the tickets to spec a project, I might write tickets to help with cleanup work on system maintenance, SPIKE tickets to investigate a new technology or technique, or "encouragement" tickets to nudge the team in a new direction. I try to be very clear why I'm writing a ticket, but having potential work documented has made it much easier for EMs and teammates to understand what outcome I'm expecting.

Design Docs

One of the bigger surprises is that I write fewer design docs now than I used to, actually, though I know this isn't consistent depending on the role or the team. I spend more time reviewing and leaving comments or feedback. Design docs are a big way for other devs to gain visibility and develop a deeper knowledge of the functioning of the system, so I actually don't want to spend a ton of time writing documentation, unless there's a larger problem I'm trying to frame.

Weekly Reports

I've started writing a weekly "what I'm up to" so that the manager of the team I'm working on and my manager have a perspective on what's happening. This has really helped keep me focused on what's important and also let other people know what I'm up to, which can get pretty vague. I've found this really helps me feel progress in my work and also help create a positive conversation about where I can focus my time if I'm off-track.

Here are a couple of really excellent articles on the weekly summary.

Why You Should Send a Weekly Summary Email

How Do I Make Sure My Work Is Visible?

Talking to Other People

Talking to other people as a Senior Engineer was much more about being put in a room to solve a problem and then working together with other people to solve that problem. Being put in a room as a Staff Engineer means trying to drill down from a high level idea into what root causes look like and how they connect to company goals.

One thing I hear a lot as I've talked with coworkers and friends about moving up in our careers is the idea that they wished there were more opportunities just to be a great individual contributor without having to deal with the politics of it so much. Staff Engineering is definitely not that, but I also think, it has to do with the realities of what it means to "multiply yourself" as an engineer. Because "politics" is the outcome of people having different views on how to do things and the fact that being high leverage means knowing where the leverage is which means... talking to people.

In my opinion there are basically two ways to multiply yourself. You can either talk to other people and help them through knowledge transfer or decision making. Both of these solutions basically require an understanding of what the people around you are struggling with and whether or not doing something technical or informational is more important. So basically talking to people (and listening) becomes really key.

That being said being impacted by politics but not having direct control of them is challenging. We all like to imagine that there's a higher rung of the ladder where we get to not be impacted by the interpersonal weave of human society but... well... humans gonna human. I have personally found that it's better to try to turn inward and manage my emotions about challenging interpersonal situations rather than trying to effusively give feedback. Finding the right place to give feedback to affect change is key and knowing when to push is a distinction I'm still getting better at.

Support and Direction

The way I've been given and received support also changed pretty drastically. Up to Senior title often it was "where do you want to go in your career" and "what sorts of skills do you want to learn" but post role change the conversations got a lot more vague. It's both possible to "stay" a Staff Engineer but also "staying" a Staff Engineer means encountering a multitude of different problems quarter to quarter.

Voltrons are Important

Lara Hogan has this concept of the management voltron. I think it's helpful (but not sufficient) to have a great manager (if my manager is reading this, hi! I think you're great). It's important to have a variety of sources for your information and to gather different perspective on challenges. Your manager (as a staff) is going to have a very specific set of pressures they're dealing with and being able to bring an outside perspective is an important way to be useful.

Working With Managers

The other thing I've noticed with the move to Staff is how my relationship to managers changed. Firstly, at many places Staff Engineering is an equivalent level to line managers, so there's a presumption of being a peer. Often my conversations will be about helping apply specific technical knowledge to a development strategy, thoughts on how to work with and encourage engineers, and also giving direct feedback to managers of teams I'm embedded on about how they can be more effective.

Because of our vantage points within an org, giving feedback is one of the most impactful things a Staff Engineer can do.

Being a Embedded But Not a Part Of A Team

Being a senior engineer meant being a "rock" on the team. You have a depth of knowledge and understand the tools of what you're trying to accomplish. You can be called on to ship code quickly and help lead projects to success. Being a Staff Engineer meant hearing things like "we have to be ready for when you're not on the team". Given that the role reports to the "business" it means I became someone who wasn't load bearing. This changed how I think about building knowledge. Becoming fluent in a specific codebase is slightly less relevant now. It's more important to invest time so that I can quickly pick an area of the codebase up if I need. Where in other roles your goal is to train other people so you can move up, in Staff Engineering, you're goal is to train other people because "by definition of the role" you're not a load bearing pillar for the people you work with.

Doing Research

While knowledge of a given service might be less important, knowledge about the "technical system" becomes more important. Instead of being concerned about the implementation of a given pattern, I spend more time thinking about whether or not that pattern is going to fit long term, and how to match that pattern so that it works with older and new systems that might come along.

What that often looks like is understanding new technologies that the organization, other teams, or the outside world is looking at, doing enough research to figure it out, and then determining whether or not to dive further. It also occasionally means ignoring things until they become more important.

The weirdest part about how my day to day has changed is that it could be relevant for me to spend a day "thinking" about stuff in a way that it probably wouldn't have been relevant previously.

Time Management

The biggest thing I personally struggle with is how much freedom and variability I'm given in how I spend my time. On a moment to moment level there are many fewer directions (and guards) that I'm restricted in. This means I have to be thoughtful about where my energy goes. As the Staff Engineer's Path talks about, occasionally you have to do things that bring you joy even if they don't have a directly obvious outcome.

End Result

The most exciting and biggest challenge to being a staff is that the role actively asks that I spend time thinking "what would make the team better?" and then trying to find ways to insert that sort of thinking and thought process in every meeting, code review, and piece of writing I do. It's weird because it feels quite easy for this to be annoying, preachy, or unnecessary most of the time, like I'm inflicting myself on other people merely to prove I can "provide value". But at it's best, I think the sort of perspective and focus can be really beneficial.

One of the biggest narratives I've noticed (even within myself) is this idea that as you move up a corporate ladder you have more opportunities to set a "grand vision" that other people can follow. I occasionally like to joke that what really happens is you're just expanding the scope of people who have the opportunity to ignore you. But, I think the truth is somewhere in between. You want to create a big picture vision so that you can react quickly to new information, but instead of shaping the effort of the people around you, what I've spent more time doing is using my big picture theory to inform the sorts of recommendations that I might make. So the grand vision isn't an outcome I inflict on the world, it's a tool I use to help the people around me doing the work, and it's informed by the research, conversations, and practice I do on a daily basis.

I was having a discussion with some teammates the other day and they were talking about using an older monolith to stick some lookup data for retrieval across services. And I, the swaggering hot shot staff engineer, asked “why don’t you put that in a lambda?”

There was a lively back and forth but a couple of things fell out for me:

1. I finally understood the power of having compute available outside of service boundaries
2. Holy sh** Id never actually implemented the thing I was recommending (like the good thought leader I am)

It was also at this point that I realized it might be possible to not even set this up with a lambda, and instead direct connect from Appsync to S3. Some quickly googling showed me that someone had also done this https://gist.github.com/wulfmann/82649af0d9fa7a0049ff8dd1440291e4, so I wanted to dive in and try it for myself.

Overview

My basic thought was to create a simple Graphql Functionality that would accept an id and return a string with a generic name . The goal is to create the base layer of functionality, not to build a functioning system.

type Query {
  name(id: ID!): String!
}

// storage.json
{
  "record": [
    {
      "id": "d9c9e7f6-8d3d-4a5d-9d3c-1b7e3c8a8c1a",
      "name": "Tess"
    }
  ]
}

I knew that it was possible to treat S3 like a simple storage mechanism, and I'd heard that you could even do some pretty fancy stuff when it came to more complex queries from within a JSON or CSV file that's stored in S3. So I wanted to test it out for myself to see what it would look like. I'm going to walk you through how I set it up and what I learned.

If you want to skip ahead to the results you can find them here: https://github.com/Stuwert/s3-appsync-storage.

Steps I Took

1. Setting up a manually versioned s3 bucket
2. Experimenting with the query functionality directly
3. Creating a graphql api
4. Connecting the data source and debugging
5. Testing out a different pattern
6. Depreciating the bucket for one managed within the same template

1. Setting Up Manual S3 Bucket with Versioning

I wanted the JSON itself to be rather straightforward. A single file with an array of objects that stored values with an id for retrieval. I'd heard about S3 Select to query for data, so I started by manually creating a bucket in S3 and then uploading my storage file. The versioning component here might not have been that important, but I wanted to understand exactly how s3 versioned a file being uploaded repeatedly.

As you can see, I chose a very formal name.

I uploaded the storage.json file a couple of times, and noticed that the versions helpfully showed up in a tab in the UI.

I know it's also possible to actually access these versions (though probably not so important). What I liked most about it is that it makes it incredibly easy to see what versions would be live at what time, which would make debugging significantly easier in the future.

2. Experimenting with the Query Functionality

Within the Actions button there's an option to test out querying:

This would come in handy for a couple of different reasons. Firstly to help me understand how to write a valid query, and secondly to help me reference and build the query within my HTTP Resolver when the time came.

The bulk of the action lived within this console, with me testing out how exactly to write a query that gave me the results I wanted. The biggest challenge to start was trying to figure out how to get it simply to return a row of arrays from the JSON object I stored. At first I even tried changing the pattern of the JSON object to only being an array without the parent reference.

// storage.json
[
  {
    "id": "d9c9e7f6-8d3d-4a5d-9d3c-1b7e3c8a8c1a",
    "name": "Tess"
  }
]

But that just resulted in S3 assigning it a single parent value like s1_ for reference, so I knew this wasn't the path I wanted.

SELECT command - Amazon Simple Storage Service

It clicked once I found the right documentation and realized that the "Table" I wanted to reference was actually a sub-group of the object i.e. s3object[*].records . Previously I had tried SELECT * FROM s3object.records and that gave me errors. The way I've visualized this is that from an abstract perspective it's possible that there you could be multiple parent objects within the file with a records property, so the [*] is necessary to specify that we're looking within those objects.

Ultimately I landed on the following query

SELECT s.name FROM s3object[*].records[*] s WHERE s.id = [uuid];

Each [*] component indicates that we're looking in every record contained within an array.

The output it gave me was

{
  "name": "Tess"
}

What was interesting here is this keyed me in to the fact that the JSON response would give me an object that I would have to parse.

Within the request there were options for input and output formatting. Input formatting had to map to the file being parsed. So given that I stored a JSON file I had to parse it with JSON properties. However, the output formatting was independent, so I could choose to specify either JSON or CSV for the result. If I went with CSV I noticed I would actually just be given the single name instead of an object. I decided to stick with JSON as it seemed easier to parse in code later.

3. Creating the Graphql API

I went with AWS SAM to create the graphql template, though I heavily referenced the work for this CDK Gist to help set up the logic: https://gist.github.com/wulfmann/82649af0d9fa7a0049ff8dd1440291e4

There are a couple of components I want to highlight here:

GraphQLAPIKey:
  Type: AWS::AppSync::ApiKey
  Properties:
    ApiId: !GetAtt GraphQLAPI.ApiId

Setting up the API Key for the Appsync service was surprisingly easy, just a couple of lines of config.

BucketDataSource:
    Type: AWS::AppSync::DataSource
    Properties:
      Type: HTTP
      ApiId: !GetAtt GraphQLAPI.ApiId

  BucketQueryResolver:
    Type: AWS::AppSync::Resolver
    Properties:
      Kind: UNIT
      Runtime:
        Name: APPSYNC_JS
        RuntimeVersion: 1.0.0
      ApiId: !GetAtt GraphQLAPI.ApiId
      DataSourceName: !GetAtt BucketDataSource.Name
      FieldName: characterName
      TypeName: Query
      CodeS3Location: getNameComplex.js

Especially now that JS resolvers are available everywhere, getting these components set up was incredibly straightforward, just a couple of lines of config and the connections were functioning.

4. Connecting the DataSource and Debugging

The biggest challenges that I ran into were trying to format the request appropriately and then authorizing my HTTP Datasource so that it could make the request. For a variety of (probably reasonable) reasons, AWS doesn't give you a ton of details when you're running these errors, so here's what I learned.

Firstly, if you're hooking up a data source that's invoking another AWS Service, make sure you have your ServiceRoleArn, and Authorization Config set up together

ServiceRoleArn: !GetAtt ServiceRole.Arn
      HttpConfig:
        AuthorizationConfig:
          AuthorizationType: AWS_IAM
          AwsIamConfig:
            SigningRegion: us-east-1
            SigningServiceName: s3

My Cloudformation wouldn't deploy at one point and I was getting the following error The validated string is empty because I had AuthorizationConfig set within the HTTPConfig but hadn't assigned it a Service Role. The validation error is not helpful, so it's important to get these wired up at once.

The SigningRegion represents the region where you're deploying the service and the SigningServiceName represents the service you're calling out to. If that's an AWS Service you'll use the AWS Service Name, but if you're say, invoking an API Gateway, you'll actually use the name of the gateway instance, not the AWS Name of the service it's built on.

Secondly, getting permissioning and request parameters correct was a challenge. I received a variety of Runtime Error and then 403 and 405 from S3 because I didn't have permissions set up correctly or because I wasn't sending the right request. To handle an S3 Select query, it's an XML Post request.

Copy-pasta-ing from the Network tab within the S3 Bucket playground came in a ton of handy here. Their documentation is actually pretty solid, but finding something to reference was very useful. https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html

Finally, dealing with the response wasn't as straightforward as I had hoped. It actually returns a buffered string which you have to process on your own. Because the APPSYNC_JS execution environment doesn't have buffer tools (yet) https://docs.aws.amazon.com/appsync/latest/devguide/built-in-util-js.html, I went with a hackier approach and just split the string on values I determined would always be present.

You can find a reference to the full resolver here: https://github.com/Stuwert/s3-appsync-storage/blob/main/getNameComplex.js

5. Trying a Simpler Approach

Once I had the complex approach done, I wanted to see what loading the file in the resolver and then looping to grab the value would look like. https://github.com/Stuwert/s3-appsync-storage/blob/main/getNameSimple.js Frankly, it was a lot more straightforward, and didn't require hacks to manage. If it were guaranteed that the size of the json file were to stay fairly small (or be split up into multiple json files), this is probably the approach I would recommend. I sure felt smart getting the S3 query working, but given how hacky dealing with the response was, I don't think it's worth managing.

The other benefit to the simple approach is that rather than an XML POST, it's a simple GET request that parses and loops over the JSON.

6. Deprecating the Manual Bucket for a Programmatic One

From there, my final step was to get rid of the manual bucket and put one in the same template as the Appsync API.

StorageBucket:
  Type: AWS::S3::Bucket
  Properties:
    BucketName: name-storage-bucket
    VersioningConfiguration:
      Status: Enabled

It was very straightforward to do, and I could replace a bunch of string ARNs with variable references Resource: !GetAtt StorageBucket.Arn which had the benefit of making me feel like a #realengineer.

I want to call this out because I think it's easy when testing new code to go from 0 to 1000 and try to automate all of the things. It's always best to start by doing nothing. I found the combination of manual and automatic integration helpful to reduce the number of moving parts, and inject some certainty into what I was doing. Then, when I was ready to fully automate, it was a simple change.

7. Next Steps

I didn't do the final step of fully wiring this in with something like a Github action to upload the files to s3 on code change. https://github.com/marketplace/actions/upload-s3 I also didn't add authorization (which you can also implement within your Graphql Unit Resolvers).

That would take this from a toy to a functioning service, but I didn't think it was necessary to take those steps because the core of the challenge was reading stuff from S3 without a lambda.

What Did I Learn?

Live look at me building a bridge of knowledge

Paris Bridge by Arthur B. Carles, American, b. Philadelphia, Pennsylvania, 1882–1952

My recommendation got me thinking about how in the last 10 years if you needed a computer to do a thing for you (however small) you needed to find some sort of server abstraction to stick it onto, and hope that it fit well enough that you could maintain it. But with the power of AWS compute is now a “freely available” resource that you can pluck out of thin air. What this means is that, instead of thinking about problems in terms of what gravitational orbit they might fall closest to, we can think about them in terms of what is easiest for us to support and maintain, and ultimately deprecate.

Building storage-first serverless applications with HTTP APIs service integrations | Amazon Web Services

But this work actually pushed me even further to realize that the real benefits of "serverless" aren't actually the ability to push lambdas up to the cloud that live outside of big monoliths. I totally understand why, if your choices are between trying to spin up a new lambda to "do a thing" or to "add some code to a monolith" you'd go with the latter. The costs to doing the code and getting the lambda happy and dealing with cold starts are real. But... if you can decouple compute (heavy lifting that your backend does) from data access (serving stuff that your frontend needs), you can ship small and simple units of change that don't require a lot of maintenance. For example, if I wanted to move this to Dynamo in the future, it would be (I don't want to say trivial because there lies danger) a relatively simple pattern to update my datasource and swap out the resolvers doing the work, without changing the interface, or needing to manage too much of the underlying code.

The other thing I realized is that half of the reticence with my earlier recommendation is the fact that you have to stitch a lambda between Appsync (your graphql provider) and S3, your storage. Spinning up a lambda means doing stuff like managing Node or Python versions, packages, and references. Doing a direct connect is closer to 4-5 files that you could reasonably put in any repo that you feel fits.

I don't like the phrase "the best code is no code", but I've really been leaning into this idea that good code should be easy to replace and remove, and what I love most about this implementation is how small the surface area is to get rid of. If I wanted to deprecate this in the future it would be a process of removing the cloudformation stack, and archiving a repo (or deleting the files), it's a much smaller imprint than trying to pull out old code!

Hope this helps!

But on the other hand, the best relationship I've had with managers are the ones that let me express my skepticism in a healthy way, regardless of whether or not it leads to immediate change. At a previous job I was probably one of the most skeptical people when it came to the switch from Javascript to Typescript. But my manager at the time invited my disagreement and gave me an opportunity to do more research until I ended up becoming a Typescript champion on the team.

Being on the Receiving End

Look, I'll be honest, as someone who's spent a fair amount of time now trying to get other people onboard, at various places of work, of ideas that I thought were good. Skepticism can be fudging frustrating. It can occasionally feel like I'm trying to do a good thing and someone is just standing in my way out of obstinacy or a decision just not to educate themselves on what I'm trying to accomplish here, dammit.

The irony is not, in fact, lost on me.

But, at some point I have to recognize that my frustration with that situation is less a reflection of someone else's incompetence and more a reflection of my emotional attachment to a decision I've made.

What Skepticism Isn't

Being skeptical is one of the more policed forms of communication, and I think it comes down to a couple different phenomenon. Firstly, the concept of bringing your "whole self" to work. In emotionally inclusive spaces there's a kind of belief that if everyone can be their "whole self" that any single given opinion is indicative of their "true opinions". So it's less "this person is uncertain and this is how they're expressing uncertainty/confusion" and more "you're choosing to be a problem".

Firstly, I think skepticism cuts against the incredibly popular aphorism "disagree and commit". The idea of disagree and commit is that "we're a team here so once we make a decision, in order to continue being a part of that team, get on board".

Confusion expressed as skepticism is typically not obstinance in my experience. It’s someone who doesn’t know where to start and is expressing that uncertainty through critical language. Disagree and commit doesn’t work here because there’s no earnest disagreement. The issue is that there’s no material knowledge to disagree with or to commit to to.

Secondly, it's ironic to me because it cuts against the concept of the learner's mindset that Carol Dweck popularized, but it's also just not representative of the situation. Often the best way to learn is to say "This looks weird and wrong to me for X, Y, Z reasons" and then to have a conversation about why that's not the case. At least for me, trying to reframe my perspective that's not critical just isn't possible. It's not that I'm actually trying to be a jerk it's that the way my brain is perceiving this situation is through critique. And with that explanation, I can move forward with growth and learning.

But more importantly, all of these sorts of alignments are really about the end stage fof the process, not the start. The goal of collaboration on a project isn't actually to be in alignment at all points in time. It's about addressing points of disagreement and then moving forward together once a decision has been made. Being comfortable with people's "whole selves" means being ok with the fact that some days they're going to be less in alignment and some days they're going to be more in alignment. And trying to create a situation where they can be equally effective in both modes.

What Skepticism Is

Skepticism, in my experience is someone who is being open about their thoughts and bringing their "whole self" to the situation. It's often "unwarranted opinions based on knowing very little". While it can be frustrating to be on the receiving end of that, people being open about those very opinions is the best way to start a productive conversation towards future alignment.

I actually really love hearing people's unwarranted opinions now, because it gives me a clear sense of where they're at, and what sorts of information or techniques we can talk about to move towards having a productive conversation.

Meeting that skepticism with frustration has the potential to reinforce the sorts of ("command and control") emotionally unsafe behaviors we really try to avoid in tech. Some of my most productive conversations on Pull Requests have been when I say something like "this looks weird and wrong" and the developer who wrote the code walks me through what's happening.

I think it's important for both parties to extend grace in these sorts of situations. As the skeptic, I've tried to reframe my initial criticisms less as raw frustration and more as a reflection of my own lack of knowledge. But I also think it's important to respect people's skepticism (even when less than ideally worded) as a real piece of useful information that can be used productively in the future.

Actions Matter

I want to end with a reflection on the heart of "disagree and commit". I think when we hear that phrase we conjure these images of people arguing in a boardroom 12 Angry Men style and saying "I don't agree with you, but dammit do I respect you". But the truth of what actually happens is way more subtle. It probably looks more like someone leaving a bunch of comments on a PR and approving it anyway, or someone disagreeing in a conversation one moment and then shipping code related to the agreed upon direction.

Words matter, but I would much rather be around people who express frustration and still productively take action towards a shared goal, than people who are quiet and do not. The ideal is always that our words, actions, and emotions are in alignment, but we're all human, occasionally those arrows are going to be moving in different direction and in those cases it's important to differentiate genuine productive emotions (skepticism) from unproductive action (obstinance).

2 Cent Postage Stamp An 1879 Postage Stamp, Smithsonian

Optimize For the Scale In Front of You, With a Bit of Wiggle Room

Premature optimization is the root of all sins and all that. But it's not just hard to understand the shape of the usage of the tools you build. So limit your scope, ship what you understand and learn from it.

Failures are a more effective indicator of what to optimize than best practices

Failures are a reality of modern software engineering. Instead of trying to "move fast and break things" or trying to optimize until the cows come home, try to limit the scope of changes, make failures as non-detrimental to yourself and your users, and use them as part of your learning and development process.

Observability is the only way you actually understand your code

:see here

There's No Such Thing as a Free Lunch

Serverless, Observability, Test Driven Development, LLMs, Event Driven Architecture, Monoliths, etc. There's no magic bullet and they all require maintenance and upkeep to support. Invest in the tools that shift variable, unstable effort into consistent, fixed, and stress-free effort.

Ship Code That Improves Human Thriving and Decreases Human Toil (in that order)

I think too often we focus on "reducing toil" as the ultimate effort of our code. Often it's the lowest hanging fruit. It's easy enough to move things to a CI pipeline that automates X,Y,Z that lowers build times. But it's important to remember that the goal of this is to improve human thriving not to merely reduce the amount of stuff we will do (there will always be stuff, somewhere).

Engineers Provide the Most Value at the Intersection of Business and Technical Understanding

It's worth investing in both.

Invest In Your Skeptics

Complaining can actually be a form of problem solving.

Leadership Is Mostly About Managing Your Emotions and Other People's Emotions (in that order)

The first rule is to not inflict your insecurities on other people. (I am so guilty of this)

Being a "good person" is hard

I don't love the phrase "be a good person". In part because of the psychological evidence that if we identify ourselves as a "good person" we will tend to explain away bad behavior, but also because it doesn't give much of a directive on how to behave. I prefer being good to the people you're with, thoughtful for those who will come after you, and curious about the world around you.

Accomplishing something meaningful with people who care about their craft is one of the best feelings

(This is a poorly worded sentence, but it's the best I got)

I guess I should but a Spoiler warning here because this does reference a showstopper challenge on Episode 1 of the current season of Bakeoff. And… I'm sure for most people this does not matter, but I also don't want to be that guy spoiling GBB. So… anyway. You've been warned.

I had a stupid idea for an article today and I want to share it recently I shared this photo of work in a slack channel. It's an orca cake from the Great British Bakeoff. And it got a positive response. And it reminded me that leadership is mostly just stupid and mundane. (And that's a good thing)

I'm not talking about the people who liked the photo. They're all great! If you're reading this Hi, I'm so sorry for turning a lovely social interaction into fodder for the content machine 👋.

Anyway here's the photo... It's an Orca Cake

What I’ve noticed in leadership writing is two basic sorts of ideas. It's either talking about something grave and serious and honorable, you know like leaders "run towards the fire", and XYZ and leaders set vision. Or, it's about how silliness is so important to creating safety and fun. It leads to this barren (in my opinion) view of leadership that is either the most serious or the most silly and there's kind of nothing in between. And silliness and seriousness are two of the big emotional moments in leadership, but there are all sorts of smaller moments that add up to whether or not you have the social standing to be serious or be silly. And those smaller moments are the ones that count and don’t get talked about.

So anyway, basically what happened is I was watching the Great British Bakeoff and I saw this orca cake, and I was like SHARED REFERENCE so then I shared it in this slack channel with a bunch of other people who are also leaders. And I thought to myself "This is incredibly stupid and also it's in a channel with other leaders so it MUST BE LEADERSHIP." And like my brain does I started thinking about how you could turn this into some sort of leadership moment.

So I asked ChatGPT what a real LinkedIn article might say:

• "Whale of Wisdom: What an Orca Cake Photo Can Teach You About Leading with Finspiration!"
• "Baking Waves: Unleashing Leadership Lessons from the Icing Depths of an Orca Cake Snapshot"
• "The Sweet Symphony of Leadership: Decoding Orca Cakes and Team Harmony"
• "Oceans of Insight: How an Orca Cake Can Guide You to Captain Your Leadership Ship"
• "Taking the Cake: Lessons in Leadership from the Black-and-White World of Orca Confectionery"

And the point with these is that most of what you see is this highly optimized, tightly packaged narrative that gets at very little of the truth of what "leading" actually is. Which is like… a series of conversations mostly inflected by past actions, that are banal, but if managed and executed properly can in fact lead to incredible things. And sometimes it's just sharing stupid photos with each other because you're people and it's nice to enjoy work.

Like sharing an orca photo in Slack could go a bunch of different ways. The channel itself is typically focused on work things. People could be like well, you should keep this channel focused on important work related things or people could be like Stuart what the f*** are you doing like posting in a work channel after hours or it could be a symbol of my own incompetence or just like that I don't take my job seriously.

And just because it got a positive reaction like that outcome is actually the result of a lot of things that don't get talked about in leadership channels at all. It's the result of like a bunch of conversations and a bunch of work and the social standing that I've built up with group of people. It's the result of a lot of hard work that everyone who's on this team has done to get to know each other and communicate. (Some people popularized it in the framework for storming, norming, and performing)

And basically the big secret is that it's so easy to focus on the big picture emotional stuff. But all of that stuff is earned by the other moments. Having the right to be silly, depends on all of these things about your confidence, about your ability to perform for the people around you so that your silliness comes off as genuine and earned and not deflecting away from your flaws or trying to connect with people who don't trust you. Having the right to "take charge" and be listened to in the challenging moments is equally about the earned trust you get from doing the hard work the rest of the time.

I'm not saying the orca cake is like leadership, I'm saying being a leader is roughly equivalent to sharing a cute orca cake photo with some coworkers. Most of the time it's pretty banal and doesn't connect to some grand vision, but you should really try to make it fun and safe for the people you're sharing your time with. And you should make it fun and safe because it's the right thing to do, and the side benefit is that you will work more effectively together if you do that.

And I genuinely get frustrated because if you're looking at the majority of writing about leadership you would come away with the sense that it should be bright and glitzy and glamorous. But that perspective I actually think does way more harm than good. And if you have that perspective you're also probably going to be miserable because most of the minute to minute isn't actually any more or less exciting than the rest of your career was. It's just like doing small habitual repeated behaviors over and over and doing them well and consistently.

So whenever you read leadership writing or speaking, I think you should think of the orca cake. And really wonder whether what's being described is a happy outcome of a lot of hard work, or is it a habit or a principle or an idea that you can actually implement for future success.

And now, having successfully thought lead all of the fun out of this beautiful Orca Cake from Great British Bakeoff, I will end this article.

I've wrote before about the benefits of working in public (and not) and whether or not you actually have to pay attention to every (or any) trends. But I want to spend some time talking about the value of learning.

So yeah, I was in a conference room with some coworkers and we had to wipe the whiteboard clean. In a normal interaction, that would in fact be the end of this. But I am someone who writes, so I decided to write about it. (God bless you for reading this)

If you're one of the people who I was in that room with hi! :wave:

One of the problems about writing and cultures of writing (especially about learning) is that you end up spending a lot of time and energy on stuff that aren't actually about learning at all.

There are so many hacks that attract people to articles. Things like

• bullet pointed lists
• diagrams
• Large images
• Snappy concepts that can be easily repeated

Photo by Christina @ wocintechchat.com on Unsplash

But, fun fact, these things don't actually help people retain the information you're sharing. The framing is more about capturing attention than it is about sharing knowledge. It's fluff. And more than that, the types of insights that are catchy tend to either highlight or run against current trends. If you write an article that's like "Why AI is the greatest thing since sliced bread" or "Here are 15 things wrong with AI" you're way more likely to gather attention than an article about a bunch of boring, immediate changes that aren't catchy.

And that's ok. That's perfectly fine. Lots of writers try to balance the need to write things that will draw attention with actual words of actual merit within their writing. It's a real tension. But it's also not particularly helpful to mistake catchy hooks with good writing, or trying to plumb life for interactions that could be worth sharing for their business value.

You Do Not, In Fact, Have to Learn

I remember my first (and second) years of college. I attended freshman orientation, first as a freshman and then as a counselor. And I remember hearing the exhortation that part of being a good citizen meant being informed and aware of the issues. It meant being connected and in touch.

But frankly, that paradigm existed in a world where you could reasonably read "the takes" of the day and still have plenty of time left over to do your job and live. That's simply not even remotely the case anymore. And more so, the "publish or perish" model seems ever present in every industry. You have to constantly be promoting yourself so you can stand out and edge out your competitors.

This is the part where astute observers will wonder if I have any sense of irony remaining. I do. But my desire to wrench thoughts out of my mind outweighs my sense of self-consciousness.

It's really worth asking if you writing/sharing something because you think it will "gain traction" given the current moment or if you actually believe and have interest in what you're writing/sharing. Does spreading that thing have meaning to you? Or are you trying to validate some vision of being a well-connected "informed" individual.

Life, Can Actually Be Enjoyable, With Accruing Knowledge

But every action you take doesn't have to get bucketed into learning/growing or decaying. You can actually go out on a brisk walk with your dog and simply enjoy the look on her face as she jaunts along in the middle of a business day because it brings you joy. You can enjoy spending time with your coworkers because you like being around them. You can spend time alone because you want to (healthily) be alone. Not everything has to be bucketed into good or bad.

I like coming into work because I like being around my coworkers. They're smart, funny people and being around them makes me happy. If you're still reading, hi again! :wave:

It's ok to let that be enough!

Learning Is Hard

If I have (ugh) learned anything it's that learning things is actually quite hard. It requires a lot of focus, time spent not knowing if you're learning things, and ultimately seeing your knowledge in a new light before you can even begin to unpack what you've understood. That means when you decide to learn, to actually learn, you should commit to it. You should be willing to forgo other opportunities because you want to learn something from that thing.

Choosing when not to learn is important because it means you are choosing to experience and choosing to live life. There are some estimates that you have about 4-5 hours a day of meaningful focus Make them count.

My first answer was, humans are social creatures, and engineering systems especially are sociotechnical ones. (I mean sure all engineering is social too but because of the lack of accreditation/regulation and the fact that we're talking about shifting bytes of data around, it's way more social). But more recently I've come around to the idea that there are some forms of project management that are really harmful, both from a culture and also a career perspective.

I’ve come to the conclusion that engineering first cultures (both technically and socially) are the result of external pressures that align incentives where the cultural values mean that every extra unit of software/unit produced is more valuable than optimizing on the what. And this is generally the result of companies (and societies) that are rapidly expanding, have extra cash and see a huge marketplace where there’s lots of cash on the table if only they can make a system to capture it.

Everyone likes to point to Boeing as this perfect example of what happens when MBAs displace engineers. Once successful company, falls apart as it slowly becomes more profit oriented. And, yes absolutely. I too would love to operate at a company where engineering were valorized and engineering first cultures abounded. But also, finding the right alignment to work at a company where that exists is rare in both space and time. So you either have to find one of those cultures and situations or you have to orient your career to make those situations more likely… which is… once again. Sales.

It also doesn't help everyone I’ve seen writing about this is the type of person to face this problem head on in one way or another. Which I guess means that a lot of their advice about how to shape your career is somewhat impractical, both from an exemplar perspective (dunno how many of us are going to become CTOs, start our own businesses, etc.) and also from a desire perspective. If the goal is to have a job that takes up just the right amount of time and also provides some career growth, what does that mean?

I'm not sure this actually means anything to be honest, more than that the people who are writing things tend to be self-starters.

But the details of this stuff matters. There are some situations where project management (both of the tech lead and more mundane varieties) represents a real step up for your career, both in terms of responsibilities and if you're getting what they call "career growth". But also, extra interpersonal management can cause equal drag on your career the same way that doing rote software tasks can.

I think the question is rather straightforward, are you working with other people to come up with solutions and agreements around (hard) problems or are you spending your time trying to negotiate against your values.

Negatives

• Secret Performance Management (Accountability Check-Ins)
• Documenting decisions to preempt escalations and arguments
• Sizing Features not Problems to Solve

These are all pretty straightforward. If you have to do performance management in the guise of project management there's just no world where that's a career benefit. And I'm not talking about giving someone feedback on a pull request. I mean like, creating regular check-in cadences to keep someone on track, scheduling your timelines around their abilities, or having other people shift their work around to support.

Decision documentation is important, and it's important because humans are fleshy creatures who need constant reminders of the things they agreed to. That's ok. But what's not ok is needing a decision document to figure out who's to blame if a timeline gets missed or things go sideways, or creating decision documents for paper trails if things get escalated in the future. It's not that things should never get escalated, or understanding why is unimportant, but being in a culture where these things are necessities signals a significant trust breakdown that can't be hand-waved away.

Finally, are you spending more time project managing feature requests or solving problems? If you're estimating feature requests there are a number of reasons that's a losing battle. Firstly, the requester already has a number in mind, so at that point it's really about figuring out how to deliver something that approximates that. Secondly, you're going to spend a lot of time expectation managing why things are going sideways, instead of solving problems. Thirdly, the type of timing and tactical problems that come up in the course of this will not require thought to process, so you're not developing technical skills in the process of doing them.

Positives (that might seem negative)

• Writing concise (architectural) documentation
• Coordinating meetings between different teams
• (Some) Ticket organization

It would be easy to leave this discussion with some sense that I believe that documentation is bad. Far from it. But the purpose of documentation is always about point in time communication, not as an artifact. I.e. if you write something you should be trying to transmit information to another human being for a specific purpose, whether that's yourself in the future, a stakeholder or another engineer. And explaining why you want to do something, what else you considered, and your decision is a great way to quickly flex your technical chops, get other people on board, and also invite productive disagreement about alternative options.

Coordinating meetings. Often, at large technical organizations, the teams doing the work will have multiple things going on at the same time. (We're all shocked here, I know) This means occasionally you'll have to meet to check in to make sure that everyone feels confident that they can still hit the deadlines and discuss anything new that's popped up in the interim. These are good and positive meetings (and generally happen at much more infrequent cadences than the aforementioned accountability checkins).

Finally, ticket management. I actually think it's really important for developers to maintain their own tickets. The tickets are the things that connect the "what we're trying to do" to the "how we're going to do it". It is both a technical question to answer "is this important" as well as "how important" and "what are the risks". Being able to get detailed enough about the tickets that you can execute work, and also connect that work to the goal in mind is its own skill.

It's Nuanced, I Guess

The hard thing about all of this is that the benefit of software engineers are their ability to solve challenging business problems through code. But the lever to be able to solve those problems is the ability to communicate and collaborate with other engineers and stakeholders. Poor communication puts limits on how big the problems any given engineer can solve.

(Unless, once again we're talking about the aforementioned engineering first cultures, though even then you could argue building engineering first solutions with great documentation and UX is its own form of communication... but I digress)

Being able to differentiate what project management opportunities are beneficial versus toxic can help make workplaces less challenging and set you up for more success down the line.

Photo by airfocus on Unsplash

One of the intricacies of hiring is that there exists an information gap that is (now) incredibly easy to overcome, on some levels, and so the usual heuristics for measurement are harder than normal. A lot of the traditional things that differentiate Staff+ Engineers from Senior Engineers are soft skills like projects coordinated, product roadmaps supported, and standards enforced. But that's all social proof stuff that's easy enough to fake and is hard to represent or differentiate outside of direct conversations.

From a technical skills perspective how do you think about measuring a Staff Engineer against those things? And what are the hard skills that a Staff Engineer has that a Senior Engineer might not?

Things I'm (personally) not particularly interested in

Ability to create arbitrary unbounded features

I've thought about this one and this is almost always the most common interview question. But, contrary to popular belief I don't think that engineers should try to moonlight as PMs (I think engineers should generally get better at thinking in product, but that's a different story).

Secondly, this is far and away the least indicative part of the stack. Producing arbitrary feature code quickly is indicative of very few things that we actually care about in the Staff role, especially in the ways I would expect a Staff Engineer to help guide and mentor other Associate and Senior Engineers.

Shape Drawing

This is something of a personal attack (on myself) because I am in fact very good at drawing the shapes. But I'm just not sure it's a good signal to much of anything from an internal/external information perspective. Being able to quickly whip up boxes and arrows is a very good way of communicating quickly, and I would expect most Staffs and Principal Engineers to do stuff like this, but... pretty much any Senior Engineer worth their salt should also be able to do domain modeling within a reasonable amount of time as well. Being able to draw something like "Gateway->Function" doesn't tell me much of anything meaningful about your ability to map systems.

Digging into the Why

This gets back to the social proof perspective, and it's easy enough to just practice your way out of this. Asking good questions during an interview (which is something you should definitely practice as an interviewee) is good, but might not represent anything special about seniority, experience, or competence.

I also don't think being able to explain the why or the value of a particular software you "shipped" means much of anything. I'm fairly confident I can speak to at least 3 different projects at my current job that I was not involved in, and it's hard to tell how much of the answer is BS versus thoughtful engagement.

Things I am interested in

Ability to quickly model and interpret domain from reading code (across multiple systems ideally)

The thing good staff engineers can do is pick up codebases, and then understand what that system is trying to accomplish between the two and how they sit relative to each other. From there a Staff Engineer should be able to add functionality (in the right or at very least defensible location) to meet the task at hand.

Create coherent function guards and boundaries while still passing tests

Constraining a system in useful ways is a challenging but important problem that often differentiates systems thinking. Being able to do that in practical code is non-negotiable in my mind. Given a relatively straightforward function (and tests), add useful boundaries that would make system failure create more useful information, without

Quickly producing code to solve problems

I think that Staff+ engineers should be able to quickly write useful code when necessary. Especially when there are clearly defined goals (like tests or other outcomes), Staffs should be able to produce code quickly to solve those problems.

Are any of these tests Realistic?

In short, basically no. The reason why unbounded questions and "digging into the why" are so popular is that they're relatively cheap to produce and maintain. And most teams think they're good enough at sussing out BS to get away with it. The reality (especially in the current hiring market in 2025) is that no one wants to talk about all of the time you spent crafting systems to hire someone instead of you know.... crafting systems that will make your employer money more directly.

Are there elements of this you can copy?

I think so! If you're someone starting with a challenge and you've got the standard "unbounded questions" or "design a feature", I'd start by taking whatever exemplar you have as a system diagram (that you're ostensibly comparing candidates to) and share it with the candidate. Ask them to critique it and ask them what they might change and why. Then, ask them to produce the code for one of the pieces of the system, mocking out responses from connecting systems, and producing reasonable constraints on their behavior of the system.

This isn't perfect, but is getting at a few core concepts that are useful to test like how does this person respond to constraints and feedback. It also will get at things like looking at how they go from modeling a system to how they actually write code based on that model, and also what types of mocks they use to situate it. This, to me, feels like a step forward from the typical "Senior Plus" exercise that seems incredibly common.

Photo by Kelly Sikkema on Unsplash

Unexpectedly two of my favorite books in the last year are How Big Things Get Done and The Unaccountability Machine. You wouldn't really think they relate to one another. How Big Things Get Done is a book about why and how big projects happen and The Unaccountability Machine is a scathing look at why the world feels so crappy because of the way businesses and governments are set up. But both books are about systemic capacity and measurement. And both books care deeply about seeing systems not as a grouping of interchangeable parts but as a representation of processes that can not simply be decomposed without cost. This aligns with the viewpoint that the atomic measurement of software engineering isn't an engineer it's a team.

Honestly you should stop reading this summary and read both of them. They are absolutely incredible and changed the way I thought about actually completing projects. But I want to highlight a few of the big takeaways that I most appreciated.

Embedded Knowledge Is Real and Valuable

One of the most fascinating conclusions of How Big Things Get Done is that when you start a new project you cannot simply add up all of the component parts to get at a timeline. As a software engineer I cannot help but chuckle at this idea in comparison to the pointing a bunch of tickets, adding up the points and then hoping to come out with a result that resembles reality. Instead, How Big Things Get Done suggests that you must compare the outcome of similarly scoped projects to get at a reasonable expectation of delivery. You have to compare against what similar projects have actually experienced in the past.

This is very similar to what The Unaccountability Machine talks about when it talks about analyzing systems as varying degrees of black boxes. They both cannot be reduced into their component parts, and also have to be evaluated against the role which they sit. You have to map out the entire system to get useful information about its behavior.

From a software engineering perspective, this implies things like you cannot simply add an all-star engineer to make a project go faster unless you have specific knowledge about both the goals of the project and the parts of the system that they would be supporting. It also means you cannot really evaluate engineers outside of the systems they sit within. From this perspective, leveling up as an engineer becomes less about your innate technical competence and more about their ability to operate at specific levels of the system when needed. I.e. Can you operate on the org as a whole, making decisions about how its structure will change/improve based on the code that you ship? It also means you can't do things like take a team of 1 Junior and 1 Senior Engineer and replace them with 3 Mid-Level Engineers because the competency is roughly the same. You are functionally forming a new black box which you have no information about.

Adam Grant has talked about research that shows that "star employees" cannot simply transition their stardom from one company to the next when they switch roles. In fact, there's no guarantee that the stardom will persist at all. This shows two highlights that both books get at. Which implies the third thing, the knowledge that the system has is about its relationship between the parts, not the functioning of any given part. I.e. The ability for a software engineer to understand the business domain, the code domain, and delivery is not just a function of their technical acumen, but about where they sit in the business and what information they're exposed to. You can't atomize that knowledge into its component parts, even within an individual.

Stable Systems Can Withstand Lots of Errors Before Collapsing

What's interesting about both of the books is they spend a lot of time talking about the ways systems can break down, and what happens as a result. But the other takeaway I was reaffirmed of, is this idea that when big projects (or systems) fail, they tend to not fail spectacularly, outside of Black Swan style events. Since Black Swans are what get coverage, but the institutional decay that proceeds them is often very real, if unmeasured.

On a practical level there are two basic experiences I have that line up with this. Firstly, watching Twitter (X?) mostly continue to function even with a large chunk of the engineering org gone. Given that the original system was reasonably well architected, rather than seeing a broadly cascading failure like many expected, what you instead saw was pockets of instability within the system which caused bugs and annoyances, but mostly allowed the top level narrative to persist.

This aligns with my own experience of leaving and watching other leave companies. The assumption is that things will either go poof or that means that that person wasn't actually all that important. Rather, what tends to happen is that other people will compensate for a departure in a variety of ways that mostly shores up the consequences of that departure, but leaves other gaps unfilled (if that person was a regrettable exit). There, for the most part aren't flashy collapses.

In How Big Things Get Done this looks like the variety of ways that projects can fail. In the instance of the couple's renovation that went wildly over on budget and time. Life mostly persisted for those individuals in a slightly degraded fashion until they ultimately decided to leave without ever having really enjoyed their renovated home. The loss here wasn't in terms of explicit chaos (necessarily), but the valuable years of their life they left unfelt.

Similarly on software projects, it's really hard to capture the value loss of poorly run projects. Shifting engineers and product around like pawns can result in degraded working conditions, frustration in communication and longer timeline. But if a top level feature still gets delivered, who will notice the two weeks that it could have been delivered sooner? It is very hard to differentiate until someone really digs in.

Big systems can withstand a huge amount of change and mostly continue to function as-is without a huge discrepancy in what they're doing until a catastrophic event occurs, the signals being delivered in ways that never get back to central planners.

Feedback Structures Matter

Finally, the conclusion I've drawn that somewhat integrates concept from both book is this idea that if Middle Management cannot actively impact delivery in meaningful ways, they will optimize the next best thing, which is deflecting personal responsibility. In How Big Things Get Done the author talks a lot about what good project management actually looks like. Best in class estimations, stakeholder management upfront, testing and iteration on new concepts, using tried and true technologies. But what if you can't make those decisions? Well, if you're in middle management and you work for leadership that has made delivering a project reasonably impossible, what should you do? You can see the long tail coming. The reasonable/obvious options are basically either, get out or to create systems which deflect blame.

The obvious and ideal answer is, this group of people would dig in and figure out how to make it work. But this only works so long. Typically such people who do this successfully are either rewarded with promotions into leadership (of which there are fewer roles than the middle) or burn out. So what then? Basically it posits that dysfunction is the result of a system where individuals and teams cannot accurately respond to the information being put in front of them, either because they have no autonomy to do so or because they do not believe there's any value in the attempt. So the levers here as a leader might become about either reducing the volume of information coming in (i.e. reduce thrash), increasing the cost of deflection, or decreasing the cost of doing the right thing (and ideally some form of all three).

Joining this with knowledge from How Big Things Get Done creates other interesting conclusions. Basically designing a system of information to prioritize repeated attempts at similar problem sets (thereby incentivizing improvements), rewarding modularity and testing, and allowing teams to respond directly to the feedback they themselves can generate will help smooth out thrash and clarify roles and responsibilities.

Where To Go From Here

Ok so there's lots of pointing out how things are bad, what does that mean for managers and higher level contributors? I think there are two primary takeaways:

1. Feedback mechanisms matter: This is something you can poke at in 1-1s and in interview processes. Asking questions like "When things break down, how does that get resolved" or "What does accountability look like". Also digging in on questions like "How does the business measure project success?" and "What happens when problems go sideways" are good ways to understand and adapt what you're doing.
2. Problem solving mechanisms: When things go wrong, both books work in conjunction to ask a variety of useful questions. Like... "What were the incentives that created the decisions that lead to the failure?". "What system broke down? Was it immediate decision making or higher level change?" and "Who is responsible for making the necessary system design changes?"
3. As an individual contributor: It's worth asking the question, is the cost to not being able to respond to this information really challenging for me personally? (This is the definition of stress and anxiety is getting a lot of information that you are unable to respond) or if you think digging in will have a direct benefit for you? And then mapping out what would have to be true for those components to change. The options here are to remain in a system that you have no control over and functionally be buoyed by the fact that your deflection systems will protect you or leave.

Photo by Scott Blake on Unsplash

... Minor Spoilers Ahead for the Book, So Go Read It First?

And in it the narrator is this young detective assistant who dreams of running off to join the army to do the important work of defending the empire. But the book itself is an exploration of all the ways that power structures seep into our daily lives and how they shape our decisions.

I cannot tell you how deeply I feel this as a Principal Engineer. The joke that a good friend likes to make is pulling up Conway's Law whenever people ask about why easy things are so hard to do. And… yeah. Implicit power structures shape everything. Even the code we write. I’m sorry the whole “I’m here to ship code not be in meetings” doesn’t fly. Thems the breaks. We’re all in this together. Understanding and representing power structures is part of being a good software engineer. And that requires communication.

But often, we ourselves are like Din. We believe the way to use our voices has to be big. It has to be making the big speech, pushing for a big change. Shutting down injustice. On a professional level, I often imagined this looked like being the executive who got to go up on stage and "set the direction" or make the big speech. I thought those were the big important moments of communication.

And while they are important, there are other forms of more subtle types of conversation that play an equivalent, and maybe even more important role. Because, as software engineers we are responsible for so much of the experience of human life now, many of our small decisions will impact many people.

What the book shows is that the way to unravel these injustices isn’t big declarations. Mysteries aren't solved by flashy moments. They're solved by doing the hard work of picking at misunderstandings, diving into disagreements, and pulling out observations and questions in front of the right people to answer them and to take action. It’s understanding the chain of events that led to a particular decision. And then pausing. And then it’s communicating those beliefs clearly and honestly even when it might be hard. And what I mean by that is, you have a need and a purpose to continually share your opinion and your perspective with the people around you. To stick up for your thoughts, the good ones and the bad ones.

Good Software Engineering is Democratic and Democracies require Open And Honest Communication.

The tenets of the First Amendment are woven into much of the practice of software engineering as it exists today. The belief that continuous delivery is built on top of Peer Review, Open Source, and Blameless Post-Mortems are all fundamentally bound by the belief that we are all better when we share our honest perspective (even knowing our perspective is wrong and limited!) and try to learn from each other with curiosity and trust.

Look I’m sure you got into software engineering because you liked code. Maybe you liked vibe coding. I honestly barely care about the difference. Fundamentally software engineering is about understanding how technical systems impact social systems. Every line of code in some small (or big!) way is a political writ. The sum total of which operate as a set of bounds that will guide how people act with the world and their experience of it. It will determine whether or not a large and complex systems is knowable or inscrutable, and whether or not their day gets a little easier or a little harder. Your job is to not simply replicate a belief that someone else declares to you, it is to imbue the systems you create with parts of your beliefs as well.

And we do that by communicating openly and honestly. Even when it is challenging. Even when we might be wrong. Even when we might lose an argument.

But the book is a hopeful one. The bad guy does get caught in the end, even if our detectives are left to deal with the ambiguity of the outcome. There is no such thing as a perfect win, no matter how much we might try. The striving does matter. The right people striving for the right things is why I am still optimistic about the possibility of technology, in the hands of the right people.

Maybe the main challenge that Din faces in the story is that success in any single case feels so small against the immensity of the challenges facing his world. (Both physical and social) And too, in software engineering the benefit of getting one line (or even one PR!) of code right can feel so small against the immensity of the world or even the immensity of the business requirements we're trying to move ever more quickly. It is not just that occasionally our voices feel small, it's that the outcome of using them feels insignificant. But this is also missing the forest for the trees.

Not only do the small actions have direct impact, the small actions are what shed light on (raveling and unraveling) the bigger underlying structure. The number of times I've been in meetings where a single question revealed to someone that the way they'd been considering the problem would have to change and therefor open up whole new sets of opportunities is, to be honest, somewhat immeasurable. Anyone can ask these questions. And if you are in the room, you have some obligation to do so.

Admittedly none of this is always particularly flashy or eye-catching. But it is real and tangible. Code is a small nudge of the universe towards a specific set of outcomes that you, the writer of the code, deems more valuable to the world around you. And those lines of code are written in collaboration with other people, to make the code, in your collective eyes better. It is an incredible thing to have an opinion about the world and to be honest and open about it to work with others to shape the world around you.

You Have A Voice. Use It.

1. Staff Engineering is about taking up space

It's not a nice to have. It is in fact the literal job. Making yourself smaller to support others will only work so much. Not just as a matter of exerting power, but as an ability for you to actually help other people as well. Part of what people are paying you for is your ability to demand other people's attention to and to spend it well.

I made the mistake of assuming the best way to be an "equitable" staff engineer was to be as respectful of everyone else's time and decision making as possible. This often meant getting pushed out of conversations, framing opinions too weakly to get picked up, and not maximizing my impact. While I still think it's important to listen and engage other people's voices, having an opinion and making decisions about how they company (and teams) operate are important parts of the role.

2. Making lots of incremental progress is more valuable than making a bit of huge progress

This is a counterintuitive one, but given the timescales (and AI), I think it makes way more sense. At bigger companies narrative (both at the internal and external level) become way more important. Incremental progress is narrative building because it sends a message that "we're doing lots of things!" Delivering one thing particularly well, especially at firms with lots going on has diminishing returns and might even be counterproductive.

I initially thought that taking a single project through to the end was important. It's certainly how I operated at Guild. However, after seeing the results there, and getting a chance to see how Staff+ engineers operate at Cedar, I'm seeing now that leaving tendrils for other people to pick up is an excellent way to spread influence, guide the company in positive directions, and make a positive impact. While finishing a thing is incredibly satisfying, having multiple irons in the fire is probably the best way to operate at a bigger firm.

Being a Staff Engineer is about breadth as much as depth. That means starting the tendrils for 4 different things that get picked up and carried forward is actually much more valuable than diving on one thing deeply (unless that one thing can unblock or unlock a lot of people). This is a hard skill to learn especially as a software engineering when finishing things feels so important.

3. Your value directly relates to how well you further and cut against "The Big Narrative" — and the expectation is that you do both

I think this is part of the agreement that a lot of Staffs (especially ones who are exceptionally strong engineers) struggle the most with. I know I have at various points, and I'm not the strongest engineer. You both have to have an opinion about "the big thing" and also a critique of that thing that will subtly move the org forward while still reinforcing it. Right now that's AI, but it could also be a big migration, an opinion about serverless, or a belief about org structure. I've noticed a lot of people try to divide this into two camps, either fully agreeing/assimilating or fully disagreeing. I don't think this is the case. Especially with the role of the Staff, the goal is to both further the primary narrative (set by leadership) and also to influence it positively. It's impossible to do both unless you have agreeing and conflicting positions.

The nuance is that conflicting positions need to be tailored to the correct goal. Completely disagreeing with the direction is about as useful as completely agreeing (with the added friction of being completely unfun). The challenge is to be able to hold both of these thoughts simultaneously. AI has certainly been the case with this. Being "anti-AI" (I'm honestly not sure what that means in the context of engineering) isn't useful anymore, but you also don't have to be a primary cheerleader to make an impact. Finding the best way to be informed, engaged, and figuring out how to critique and shape strategy is also incredibly useful.

4. Ergonomics are the Heart of All of This

The biggest challenge I've had as a Staff (or Staff+) engineer is thinking about ergonomics in the context of getting things done. Even though I truly love UX it's hard for me to get in the mindset of making things easier for other developers. Often I'm completely focused on the problem in front and thinking about a slight tweak that would make life more efficient for other developers.

It's the "Staff" part of Staff engineering and it's a skill I'm still in the process of building. Because you're leading with influence, making something "good" is often not a useful barometer for how well it gets picked up. The ergonomics (how easy it is to hit the ground running) will often determine if a tendril gets picked up by another developer and carried forward.

5. AI Complicates All of This

AI trades coherence for speed. LLMs trade coherence for speed. Which is, incredibly interesting. It's a fact that enables a great many things, regardless of what you think about it. And it draws to question how much coherence is important when it comes to certain areas of product management and software engineering. From a writing perspective, where the point of writing is coherence, that tradeoff is poor. Writing something incoherent has minimal value.

From an engineering perspective, if you can make that tradeoff in ways where you can increase cycles with direct goals to improve coherence (or at least enforce it), that tradeoff can be incredibly beneficial. But, ironically, one of the primary goals of Staff Engineering is coherence at the org level, which means that being a staff gets much harder with all of these tools, because you have many more actions going out in many more different directions that you have to make decisions about how to pick up or drop. And those actions are getting less coherent.

Silicon Valley, and software startups generally, have always prized speed over coherence, but there were reasonable limits on those tradeoffs (the humans who had to produce the code and the complexity of the state machines which could auto generate it). But that's mostly disappeared, and we're currently in the process of figuring out what if any hard limits there are. In a practical sense that makes picking what threads to carry forward, and how to influence people on those threads much harder.