January, 2021 LinkedIn, Twitter
Tell us a little about your current role: where do you work, what’s your title, and generally what sort of work do you and your team do.
I’m a principal engineer at Stitch Fix which is an online personalized styling company.
The team I work for right now is a core engineering team that supports our customer facing application organization. The team has a range of folks with a broad array of experience including native mobile, frontend, and backend experience.The work we do depends on the project and can include internal consulting and mentoring from design to architecture. We also manage a number of shared libraries and services. I tend to fall in on the API and backend side of things. For instance, one project I’m working on is our roll out of GraphQL and GraphQL Federation.
To date, our engineering organization has organized around particular user groups like our customers (whom we refer to as clients), our stylists, or our warehouse folks. The team I work on was originally created to focus on our customer facing application teams. While a large part of our time is still spent on that work, we’ve recently moved into our platform organization and I imagine our scope will change over time.
You mentioned shared libraries, architectural guidance, and also owning some core services. How do you decide what to focus on at any given time?
I’ve never found one way to do this kind of work. For the most part, it feels team specific. The way you might determine your highest priority work on a mobile team is quite different from how we do it on a platform team. But there are a few similarities in the way I try to approach this task, and I think this would apply to all my roles at Stitch Fix.
First, I try to ground myself in three things:
- The goals of the business.
- The goals of my team.
- The friction involved in iterating towards our goals.
I ground myself because the process is organic and fluid. It’s an alignment process which can involve difficult conversations. Those conversations are better when I’m an active listener. Being grounded lets me ask questions that illuminate how others are trying to achieve the same goals. Ultimately, that helps me align with my team and it can help align the team with the business. At least, that’s my aim. There’s never a perfect way to approach these conversations, but I trust my team’s intentions. I know everyone wants the best outcome quarter in, quarter out.
One difference between a platform team and a feature team is gauging our impact. It’s a broad purview and it’s not always clear how to best make that impact or even how to measure it. Again, I find that being grounded helps, but I’ve got a lot to learn. Even as I try to be active in these discussions I rely on my manager and my teammates who have a lot more experience than I do set the final priority list.
Some of Alex's writing
How do you, as a developer productivity team, avoid getting detached from the core development process used by the engineers your team supports?
While part of our mission historically has included developer happiness and productivity, it also is in part about supporting technical excellence. Regardless of the mission though, when you work on a team that supports other engineering teams you must build empathy for their experience. I’ve found two approaches that help in different ways, one is more like product development, and the other is old fashioned one on ones.
The best product folks I work with have a deep understanding of the users experience. In a number of domains building that understanding can take a long time. Especially when you are building software for expert use. Luckily for me, I’ve done the job. I hope I have a strong understanding of the landscape. More importantly, I can spend time working alongside folks who are shipping features. I enjoy the pendulum between feature and platform work, it’s incredibly rewarding, and I think it prevents me from getting out of touch. I bring this approach to our team’s work as well. Using a product approach we are uniquely qualified to support our engineering teams.
Another thing that helps keep me connected is 1on1s. I’m fascinated by all the folks working at Stitch Fix and how they approach their jobs. I learn something new every meeting. I try to meet regularly with 10-20 folks. The cadence is anywhere from once a week to monthly. It tends to work out so that I have 1 or 2 a day. I have found that strict agendas can get in the way, so I keep it agendaless. Listening, getting to know folks, and building a sense of their experience is my goal. Frankly, 1on1s are invaluable and while I think they are particularly valuable for my work, I would do them regardless.
Besides putting me in touch with different viewpoints, I find it’s easier to give and get feedback when 1on1s are on a regular cadence. Especially when we’ve missed the mark somehow. I want to know ASAP when we’ve made a strategic error. I’ve found regular meetings with folks speed up the feedback process. Folks don’t have to wait or worry that they are invading your schedule to give feedback. On top of individual feedback, 1on1s allow me to contrast folks’ experience around the organization. Building tools for a wide audience requires understanding competing needs. The best way to build a rich mental model of work is to hear from the folks themselves.
What does a “normal” Staff-plus engineer do at your company? Does your role look that way or does it differ?
At Stitch Fix, we have software engineer, lead, principal, and architect. We have a rubric for promotions. Which describes what we expect from an IC at any specific level. The rubric focuses on a number of things, but impact and influence are important. As you progress in your role, your impact and influence should expand. Although the expectations are clear, the approach will vary.
I see principals and architects having a significant role in quarterly and long term planning for our technical systems. They also tend to work more horizontally. With Architects spending more time on the big picture. Like initiatives that might require a year or more of time across multiple engineering divisions, things that require connecting the dots over a longer time span.
In that sense, I definitely feel that I work more in a horizontal capacity than I did previously, and also try and contribute to the big picture stuff when I can.
How do you spend your time day-to-day?
I spend my time in three broad categories: API Infrastructure, Service Operation, and Reflection.
Right now a large part of my day to day is supporting a migration to GraphQL, which is new for us. One way I support the roll out is by making myself available. This ranges from ad-hoc pairing to design consultation. I keep large blocks of unplanned time so that I can be available for that kind of work. Sometimes I’ll embed with a team for a quarter or more, especially if that team happens to be working directly with an infrastructure change I am working on.
Another broad area is working on technical implementations. As part of our GraphQL rollout there are some shared services. Those services are currently owned by my team. We are also principal contributors to a number of our GraphQL tools. There are other areas of technical implementations I contribute to as well, but GraphQL currently takes up most of my time.
The last part of my job is trying to think broadly about our organization and the future. Integrating my own experience and what I hear from others requires reflection. In my case that means a lot of reading, writing, analyzing, and thinking.
My goal, much like the OKR process is to be in a position to propose long term goals and practical methods for achieving those goals that are grounded in our current organizational strengths. I hope to produce a long term vision that expands beyond any given quarter.
A big part of this process is working with others on my team. While I make a modest impact by myself, working with others is when we start cooking with gas. I appreciate working on this with my teammates on this. I don’t always make it easy, I’ve got an opinion or two, but I love it when I’m able to broaden my perspective.
There’s a fantastic blog post, How Big Technical Changes Happen at Slack by Keith Adams and Johnny Rodgers about how they make technical changes, and defining interfaces is one of the most important kinds of changes. How did you decide on GraphQL?
We adopted GraphQL at Stitch FIx because it fit our particular needs and allowed us to work the way we want to. Although the process for achieving that alignment is a bit more involved.
We have an interesting framework for making big decisions like this. I’ve never seen this kind of framework at any other company that I’ve worked at. It’s called A Framework For Responsible Innovation and was created by one of the senior architects, Coraline Ada Ehmke. Coraline worked with the senior management to create a way for anyone to propose something new, and speak to the business value of it. It suggests starting small and over time, building a bigger and bigger business case for any innovation.
That was the process we used for GraphQL. I see technical investment as a way to build capacity over time. Always ask the question: are we delivering more value this quarter than last quarter? As we try to deliver value, we experience friction, it’s not always easy to articulate, but often with enough time and practice you can zero in on the major patterns.
After a year or so of working with the iOS team, it was clear we were experiencing friction around our APIs. The iOS team had been at this for a year or more before my time as well. Because of our experience trying to deliver value, we were able to easily identify a number of ways that our current API strategies weren’t working for us. As we looked back on that friction, and other things like post-mortems, It was clear to us that a RESTful based approach was at the top of the list. With that assessment framework in hand we explored a few new techniques. Based on our future needs and past experiences GraphQL seemed like the best choice.
As we began to migrate the iOS app to a GraphQL backend, we realized that it might be valuable for all of our customer facing applications. We worked with the larger team to grow our scope to an organizational level and we put more resources behind it. It has continued to grow in scope over time as well due to the investments that are being made by our senior ICs and leadership.
Where do you feel most impactful as a Staff-plus engineer?
Foremost by viscerally understanding the experience of a developer. In order to advocate for them and reason about large tech investments. Especially taking the time to understand how engineers will utilize that investment.
For instance, imagine you are trying to ship, and you run into a situation where you want to trade what you think is the right thing longer term, for the get it done now thing. Maybe it’s something really cringeworthy. Furthermore maybe the right thing is in a far off codebase. It’s going to take your hours if not days to “do the right thing”.
These are the kinds of experiences that I find frustrating, and I think others do as well. I want to understand what it’s like when a developer is making those trade offs. And, I want to improve their capacity to deal with these kinds of situations.
By taking the time to have that visceral sense of the work, I think we can make the right trade offs at the platform level. That might be the biggest impact I have.
One common technique for identifying pain points is developer experience surveys. Have you found that technique valuable?
Yes, surveys can be an important tool in understanding impact. But, I think they are only one part of the puzzle.
This year I’ve tried to dig into the practice of gathering qualitative data in general. When this kind of work is done well, it’s instrumental. The work that Nicole Forsgren has done with the State of Dev Ops for instance is incredible (she gave a great talk about how she creates survey questions). This is another recent example that I really appreciated. “Daily Stand-Up Meetings: Start Breaking the Rules”. The authors of the paper take on standup meetings, maybe one of the most sacrosanct rituals in a teams day. In it they use a large amount of qualitative data in order to produce a few surprising recommendations.
What I appreciate about both of those teams is that the quality of the outcome relies largely on the author’s deep understanding of the phenomena they are investigating. It’s that understanding that makes their investigations so valuable.
When it comes to my team or my organization We should have the same level of understanding when it comes to the teams we support. It makes our work valuable to our organization in the same way that the devops report was to the industry. I want to have a visceral understanding of what it’s like to do the work. It can be difficult to do that though. An uncomfortable, but not so surprising fact keeps showing up in the papers I read. When developers receive surveys, or are interviewed they tend to see it as an evaluation of themselves. Teams, organizations, and systems are already hard to understand and manage, when you also add on top of that a widely held skepticism of introspection, it can be hard to say the least. Spending time and building trust that’s where it starts though, surveys are a part of the equation, but if you aren’t building those surveys on top of a strong foundation I’m not sure they will tell you much.
Lastly, I think surveys and really any qualitative research takes a lot of time and effort. This is borne out by the research, and my own experience. It’s possible there might be a different way. If an organization is aligned well and has certain shared values that lead to a high trust organization you might be able to forgo the costly and time consuming process. In either case, the work is super valuable.
Many Staff engineers are so good at identifying problems that they have far more problems than they can tackle. Once you’ve found a problem, how do you go about addressing it?
In general, people are usually trying to make things better. But, knowledge and experience are often siloed. That makes it difficult to readily understand the impact of a problem, how universal a solution may be, and the value of solving it.
This is another reason why we must have a product mindset. Hopefully through a product mindset your team is aligned on the outcomes you are looking for. Hopefully, through some mechanism your team is aligned on how developers work. So, the next step is triaging the list. It’s another alignment process. The process is valuable, while also being one of the most difficult you can have. At the end, you should be able to say the thing at the top of the list will have the biggest net positive impact.
Are there any particular metrics or approaches you’ve found particularly helpful for explaining why a specific technical investment should be a business priority?
It would be nice if all our projects had a simple quantitative measure. Sometimes it’s easy and straightforward, but most times it’s not. For me it always depends on the specifics of a project. Which means the way I build a case is different for each project.
In general though, I’m always trying to understand how a project will impact the business. For me that includes the positive impact and potential negative impacts. Risk and incidents tell as compelling a story as new feature work. It’s not as fun to talk about though.
When it comes to the most amorphous kinds of tech investment. Things like, “architectural tech debt’, or replatforming, etc. That’s when it really pays to have a strong understanding of how work gets done. Being able to explain the experience of doing the work, and then being able explain how the investment will change that experience is powerful.
Can you think of anything you’ve done as a Staff-plus engineer that you weren’t able to or wouldn’t have done before reaching that title?
Because of the responsible innovation framework, it’s possible for anyone at Stitch Fix to suggest and try anything, with the right alignment and justification. It’s empowering as an individual. I was able to try some things when I was a lead, that would have been difficult at other places. Though, when it comes to trying something that involves more than your team, it can be easier with a title.
The biggest impact was personal, and maybe this wasn’t how it should have gone. When you don’t have the right credentials it’s easy to fall into the trap of proving you are capable. And, frankly, doing the work and being recognized is one way to relieve the pressure. Now If I have a hunch that something is valuable, I’ll give myself more time to figure it out. I feel more permission to spend time in the ambiguous areas.
The net effect is a sense of time scale I guess. Some things are gonna take a long time. Large changes in systems take time. It’s easy to think that all good things should happen immediately, but that was often a false sense of urgency. Now certain kinds of work can unfold over time, and that’s okay.
How have you sponsored other engineers? Is sponsoring other engineers an important aspect of your role?
It’s something that I personally want to work on a lot and it’s, it’s something that I’m trying to get better at. I try to make it very clear the outcome that we’re looking for, like a vision. Then I tend to start on the most important piece and make sure that that gets done.
I find that if folks can share that vision they’ll start to come along and say like, “oh, hey, can I help, can I do this”. That’s where I will try to take a step back and help instead of doing the work.
Another way I’ve found to help is often folks will identify an outcome they want to achieve and I can help them work through that process.
One way I want to be better is my ability to help folks who may not feel the authority to step in. I think there’s a lot of people who can do the job, but may not feel comfortable stepping in and doing the work. The way that I’m working on this right now is to find or create spaces where folks feel the support they need to step up. I don’t feel like I have a great answer yet, but I’m working on it.
Is sponsoring other engineers harder as a remote engineer or in a remote engineering organization?
In short yes, but, it’s complicated. When I worked in an office, the predominant pattern I saw was informal sponsorship. It relies almost entirely on happenstance. That meant that there were a lot of ways in which sponsorship wouldn’t happen. It’s yet another way we’ve systematically disenfranchised folks. My hope is that with remote work we might find ways to bring a new level of rigour to the process.
This is something I appreciate about the remote culture at Stitch Fix. We have a lot of folks who are experimenting with new ways of running shared spaces that attempt to level the playing field.
A few examples I’ve seen are:
- Finding a visual way to keep track of who’s in line to speak
- In large group presentations, using tools create a queue of questions, instead the regular free for all.
- Reducing the size of meetings to encourage contribution. I am particularly fond of a saying from my coworker Chase Clettenberg: “More than eight don’t collaborate”.
I appreciate these things because they bring a whole new attitude to our shared remote spaces, one that is a little bit more aware. I hope that this level of rigour will help remote work leapfrog non-remote work. And in the future we can say that remote sponsorship is as easy, or better than in-person.
What was your path to attaining the Principal title at Stitchfix?
I joined Stitch Fix as a lead developer working on the backend for our iOS application.
I dove in and did the job delivering features through the API. During that time though I experienced friction while doing the work. As we delivered features, we would also spend time and effort to reduce that friction.
We tried to reflect on all the reasons for that friction. We used retros and post mortems to identify perennial issues. We talked through the possible systemic reasons for the friction.
While working on reducing the friction in our system we found patterns that were applicable more broadly, so I would also spend time working with other teams to see if they found our work valuable. This led to a natural expansion of my duties.
At the same time I was working with my manager on the promotion process. We spent time looking at the rubric to understand what was expected of a principal developer and we worked together on a plan to move from lead to principal.
While reducing friction led us to trying out GraphQL. I think a number of the projects we undertook to reduce friction were a key part in my promotion. Specifically, we worked on friction that was not only directly affecting us as developers, but it was also affecting the delivery of value to our customers.
There is a popular idea that becoming a Staff engineer requires completing a Staff project. Was your work on GraphQL considered a Staff project for you?
Here at Stitch Fix at least that isn’t a specific requirement. While I think that some projects can make it obvious that you are meeting the expectations and are ready for promotion, it’s not required. I’m not sure I’ve ever seen that as a requirement. There are a number of Staff+ folks I’ve worked with in my career whose influence is less about delivering big projects and more about nurturing or helping others do that work. Both things are valuable.
It seems like StitchFix has done a great job of valuing what Tanya Reilly calls being glue. What has StitchFix done to create that culture?
There feels like a lot of room at Stitch Fix to create a narrative. Not all issues you run into day to day can be fixed in an hour. Sometimes you need to collect all the pieces and then arrange them in order to draw attention to a particular issue over time. When you zoom out it’s possible to connect the dots - that thing that looked kind of small before, may now seem like a bigger issue - And so I think that with time and space you can bend people’s attention to focus on the big and small things that have an impact.
What advice do you have for someone who is a tech lead now and wants to make the transition to Principal?
Actually, funny enough, I was just reading your guide recently, “Work on what matters”, and that’s what I would say to someone. Earlier In my career, I built things that were so far from the mark, like not even close. That hurts. I chose to work on it. Nowadays, I get a lot of positive feedback that I am spending time on valuable investments. I owe a lot to recognizing when I was far from the mark and then working on improving my ability to build the appropriate thing.
It felt a lot better when I started to focus on what’s important to the company, and what’s important to the user base, and then trying to bridge the gap by making services and experiences that were beneficial to all parties. I realized that if you can focus your work on that nexus, If you can reduce friction and increase the capacity of your organization to continually deliver at that nexus, you’ll never be far from the mark. It feels like a clear way to progress. You’ll learn a lot and you’ll also make a big impact.
In my interview with Michelle Bu, she warns against pursuing the title before you’ve made sure you’d enjoy the work, which I thought was an important point. It’s easy to look past the work and only see the title.
Interesting. I’ll take your interview and raise you a blog post Charity Majors wrote. Her thesis is that you shouldn’t see managing and IC work as a binary, instead there might be a lot of value in moving back and forth between the two. The goal is a shift in perspective.
That shift in perspective feels like it can apply to things other than moving between management and IC work. Hopefully your organization is creating opportunities for you to try out new kinds of work. If it doesn’t you should consider doing it yourself. Find ways to shift your perspective in order to see the work in a new way.
So, in that way, I agree, organizations should help folks find work they think is rewarding, and you shouldn’t necessarily pursue a title unless it’s rewarding. This can get messy in organizations where pay is connected to title though.
Finding the nexus of company need and impact makes intuitive sense, but I’ve found that some folks conflate what they’re interested in with what the company needs. Do you have advice for folks on how to avoid doing that?
Yep, I’ve totally done that before. But, I’m not sure it’s avoidable. I’ve worked with folks who can learn before they leap. I’m not one of those people. I had to do the wrong thing in order to know what the right thing was. Overtime, I feel like I’ve come to understand that and can do the work upfront.
Looking back though, what I wish I spent more time on was clarifying the value proposition. Like, constantly. What is valuable about this for our users, etc. Then zeroing in on that. Often, when you do that technical considerations start to drop away. Obviously, over time a successful business will become more complex and you’ll need to invest in technical solutions that increase your org’s capacity to deal with it, but even in those situations you should still be focused on the value to be delivered.
When it comes to value delivery there are a few resources that I appreciate:
- The Goal: A book about value delivery in a physical manufacturing plant (The Phoenix Project is a re-telling of this story in IT)
- DevOpsCafe Podcast: Go through the archives. Even if you are a developer only. Many of the guests have been involved in large scale change and have valuable lessons for listeners. Also, its a great companion to all the Dev Ops books out their.
- Wardley Maps: Connect your user outcomes to your technical components in a clear way. The concepts by themselves are incredibly valuable when it comes to weighing where you invest.
I’m going to come back to this idea one more time. Having a product mindset is the best way to ensure you are working on the most appropriate thing. Always get to know your users. What do they want? Then ruthlessly think about your time and effort as investment credits. How are your choices helping deliver value to your user.
What about a piece of advice for someone who has just started as a Staff or Principal engineer?
First, make sure you continue to advocate for the folks who are doing the hard work. That’s tougher than it sounds. When you move farther away from the teams doing the work there’s a tendency for trust to drop. When that happens you won’t have a clear picture of how work actually gets done.
Second, understand the bigger picture, and how the work of ICs fits in. Share that with anyone who will listen. Help folks understand how they fit in. There’s a lot of anxiety in our industry around one’s capabilities. Imposter Syndrome, et all. It’s hard to connect the dots between what you’re working on and it’s value to the company. Folks find it hard to see how they are fully capable, but there is this stuff called computers and organizations that stand in their way most of the time. Senior IC’s make a big impact by shining a light on effort and complexity that’s hard for everyone to see. If you can help folks find value in their work and recognize capabilities they already have you are well on your way. At least, I have to hope so because that’s my plan.
You mentioned becoming a manager through happenstance, but did you ever consider deliberately taking on an engineering management role?
As a manager, I would mandate the solution. Oof, that was horrible, a big mistake. As an IC I must build a coalition. It forces me to prove the value. While I hope I’ve learned my lesson, I’m not in a hurry to find out. I’m quite happy being a coalition builder.
Also, I don’t see the work of a manager as fundamentally different from an IC. We’re all trying to deliver more value to our customers. In an organization, there are many functions that need to get done for that to happen. Some of those functions are assigned to a manager, and some to ICs. For me, I don’t feel like there is anything that prevents me from impacting or influencing any of those functions in order to improve delivery. Someday, if it feels valuable, I can see taking on a new set of functions directly. But, especially at Stitch Fix, I’ve seen the impact our senior IC’s have, and I don’t see any limits.
Probably the number one email I get about StaffEng is that folks want to amend the Staff archetypes to include a Staff engineer whose sole contribution is writing amazing software. Is that an archetype that you’ve worked with or seen?
If you mean a person who disappears for months at a time and when they come back drops technical improvements on the rest of us that work and solve the real problem. Then, no. I’ve seen people try. It’s possible that I’ve tried that, and it doesn’t work.
What are some resources (books, blogs, people, etc) you’ve learned from? Who are your role models in the field?
I’m a huge fan of the work you’ve done. Your blog post about launching Digg v4 is a poignant reminder of how it’s always possible to get into a stomach turning situation. Digg was a big deal when I was younger. I remember watching Kevin Rose talk about Digg on Tech TV. It’s always a joy to read about how folks made those early services go from the inside. I find myself regularly sharing stuff from StaffEng and your blog.
Your experience shows in your writing. That means a lot to me. It’s clear that you’ve done a thing, reflected on it, and then shared what you’ve learned. When I decide how much stock to put into something, that is the number one quality I am looking for. How does this person know what they know?
I think there are others out there that share that quality.
Nicole Forsgren, PHD - She was a sysadmin for a number of years before earning a PHD. Her work is grounded in that practice and it shows.
John Allspaw - I remember the first time I heard about how often Flickr deployed. This, at a time when CI/CD was novel, blew my mind. Equally impressive was that while he was CTO of Etsy he earned a masters degree in Human Factors and Systems Safety. Anyway, I highly recommend keeping up with whatever he’s interested in. When it comes to the next 10 or 20 years of our careers I think he’s on it in terms of growing the capacity of individuals.
Kelsey Hightower - Ostensibly he’s a developer advocate, but he’s got such an expansive vision on the future of infrastructure. If you’ve ever dealt with the bowls of things like the AWS CLI or Cloud Formation you know – there be dragons. Speaking of capacity, when you ask, how do we grow our capacity to deal with the cloud, I feel like Hightower has an answer.
I find that it’s not enough to have purely technical recommendations. I personally find a lot of inspiration from books and tv.
Robin Sloan - a fiction writer, crafter, olive oil maker, modern renaissance day man. His writing has an incredibly organic feel for speculative fiction. As a previous bay area resident it feels like he’s soaking up the ether and transmuting it into witty commentary on tech. Start with a slightly older piece stock and flow, and a recent piece Berkeley and its casual geniuses. Also make sure to checkout, Mr. Penumbra’s 24‑Hour Bookstore, the story. I ran across this short story in my feed reader many years ago, but I didn’t realize it was fiction. I believed I was reading a first person nonfiction narrative. Needless to say, at the end of the story, for a moment I believed that magic was real, before snapping back. It gave me chills.
Kim Stanley Robinson - SF Writer. The reality of the next 100 years lies somewhere between two extremes. I’m not sure if humanity will meet it with grace at all times, but I believe in the ingenuity and resilience of us all. I feel like Robinson writes particularly rich accounts of humanity, while also embracing our complex systems. I’m reading The Ministry of The Future right now.
David Simon - Journalist, Author, TV Producer - Watch The Wire. Looking back, this show was my earliest exposure to the idea of systems thinking. He has a knack for understanding this kind of stuff, and as far as I can tell, finding the right folks to help him tell the stories.