October, 2022 linkedin
Tell us a little about your current role: where do you work, your title and generally the sort of work do you and your team do.
I’m a Principal Engineer at Google. In my day job I lead a team developing web infrastructure for internal enterprise applications. We develop shared UI components, make deploying new applications easier, and coach teams on frontend architecture and accessibility. Google relies on 1000s of internal applications to power everything from our Code Review processes to our financial systems, and from our physical security systems to continuous integration tooling. Often these systems are built by smaller full-stack teams that have to figure out how to harness some very powerful, and complex infrastructure to solve Google-scale business problems. My team’s job is to provide tools and libraries to enable the creation of really great webserving options and great UX for our internal systems.
Oh, and because this is Google I have a 20% project in which I am the tech lead for Google’s general engineering practices documentation (testing, code health, code review, handling build breakages, design doc guidance, etc.). Mostly, this job is about learning how to craft policy and guidelines that can work for tens of thousands of opinionated engineers across thousands of problem domains.
What does a “normal” Staff-plus engineer do at your company? Does your role look that way or does it differ?
At Google, Staff Engineer is the first job level where you are accountable for business strategy. Prior to Staff, you can be largely successful just by building good technical solutions. If the project fails for business reasons, it is rarely expected that lower level engineers are responsible. Staff engineers, however, are judged by not just the technical execution but by the choice and framing of the problems to be solved as understood by the business.
My favorite explanation of Staff Eng at Google (which is Level 6) is this: promotion to levels 4 (SWE III) and 5 (Sr. Eng) are largely about doing more individually - getting more efficient with tooling, faster at problem solving, and learning to write piles and piles of code. At L5 we start to ask for signs of leadership (basic Tech Lead stuff) but on the whole, the skill difference for levels 3 through 5 is linear (steep, sure, but linear). L6, or Staff, is more like a change in job ladder. Staff Eng at Google shares far less transfer knowledge from previous levels - skills like writing code matter less than clear communication and ability to think strategically.
Google’s Staff Engineers are expected to solve mostly open-ended problems where solutions are not-at-all clear to very senior people who need them solved. At a company of our scale, Staff also means a lot of coordination (read: meetings, docs, and emails). Google also makes a special point of calling out the importance of Staff engineers providing mentorship to more junior folks. Perhaps the quality most often cited during promotion attempts of Staff engineers is the “ability to reduce chaos from the system”. Which means their technical solutions actively reduce complexity and technical debt. Staff engineers are expected to model excellent engineering practices with a focus on improving Google’s overall technical health.
Having been Staff+ for a number of years now, I think this explanation is a pretty good match to my daily life, though keep in mind Google is an enormous company and this is just one perspective.
How do you spend your time day-to-day?
Lately I have spent most of my time on a lot of meetings, doc writing, and thinking. It takes a lot of communication to keep large projects on track and ensure everyone working on them (UX, PM, Eng etc) stay closely connected. I work on internal infrastructure so it is very engineering-driven which means a lot of the vision and strategy starts with me. I also do a fair amount of mentoring. As far as I know mentoring is the most effective way to actually grow better engineers. I try to maintain at least one or two active mentoring relationships at times.
Because I work on fairly large projects that span many teams I have to spend a lot of time writing - writing docs about our strategy, writing docs about values, and of course docs about system designs. When I’m not writing docs, I’m reading them - both to review and to learn what’s going on around the company. I’ve found you can really scale your own impact if you can get better at written communication. I’d like to think I’ve improved as a writer, which is a good thing because Google Docs is basically my IDE nowadays.
Where do you feel most impactful as a Staff-plus engineer? A specific story would be grand.
I think my super power as a Staff engineer has been working across teams to make connections that might otherwise be missed. In a company as large as Google it can be really common to find the pieces of a solution to the problem you are working on already exist. However, because the pieces can be very far apart organizationally, the people working on them may not realize anyone else is working on related stuff. This isn’t a problem that just Staff-level people can solve, but we are often at the perfect place in the organizational hierarchy to have cross-org visibility and still be close enough to the technical problems to understand how to put the pieces together. A level or two lower and you’re too focused on your local problems, a level or two higher and you can easily get lost in managing an organization. Staff really is the perfect level to get into trouble :).
A good example of this comes from my current project where we have built a new system to more rapidly create and deploy web applications. When we started the project we knew some folks in Google had done similar work in the past and there were partial solutions laying around, however, they were all command line tools. We wanted a web application to drive the process. At first we were convinced that we had to build our own, however, I happened to hear about some random web app that was being used to create ’low code’ applications in our global business unit. I went and took a look and I could immediately see about 70% of the tool we wanted to build. I then spent a quarter or two working with SREs from another org to connect the service that powered one of their command line tools to this web UI and I could pretty quickly see that it was going to work. I was then able to put together a team to help me realize the vision. Altogether, there were bits and pieces from our Ads SREs, our Core Web infra, a Global biz team, and then my enterprise web infra team. I had to spend a lot of time building the relationships and connecting the various teams to each other. It took some time, but in the end we didn’t duplicate anything and were able to integrate best-in-company solutions.
Without the experience and visibility I had as a Staff engineer, I think it would have been hard to find the opportunity and recognize it for what it was. In a large company it can be really easy to let your organizational boundaries prevent you from seeing what’s possible and sometimes being in a more senior position can help you peek over the wall and see what’s going on next door.
Can you think of anything you’ve done as a Staff-plus engineer that you weren’t able to or wouldn’t have done before reaching that title?
It’s probably less about the specific title, but there are definitely a set of skills I have honed over the last 4-5 years that I would not be able to do my current job without.
- Comfort with conflict - once you get to leading teams of a good size, or you are working on high pressure projects, conflict will come up. I was not as good at dealing with that as I am now. I used to beat myself up anytime someone was unhappy with me and it was really painful. I still don’t like making people unhappy, but I also have more courage and confidence in my abilities so I’m better at not letting it get to me.
- Seeing things from a systems perspective - Some might call it being strategic, but I like the idea of systems thinking. My job is now as much about how to solve today’s problem as it is understanding what might be a problem tomorrow. I need to understand what trends are happening inside and outside the company and figure out how that may start impacting my goals over the next N years. The corollary to that is I need to be able to take a very long view of success. For example, when migrating a fleet of 100s or 1000s of applications from one UI framework to another, success takes 5+ years to land. I have to pace myself and find a way to appreciate the progress we make during the journey.
- Understanding the role of values and culture - One of the challenges I had to wrestle with as I started growing at Google was that I couldn’t make all the decisions. I really really wanted to, but I just became a bottleneck. I’ve had to learn how to develop team-level values that create a specific culture that ultimately produces the kind of outcomes I want. I had to look way up stream from the problems I was seeing to recognize that the incentives and values I created for a team were the key to scaling my technical philosophy.
- Project management - I have found that being a project management geek has helped me understand how to navigate long running projects with high degrees of coordination.
These skills have allowed me to take on much bigger projects with much longer time horizons than I ever would have been able to tackle before. When I started at Google 10 years ago, I would have been happy planning a project that lasted 6 months and landing it the way I wanted. Now I’m comfortable trying to figure out what a 5-7 year effort will have to look like.
Do you spend time advocating for technology, practice, process or architectural change? What’s something you’ve advocated for? Can you share a story of influencing your organization?
I consider myself a full time advocate for a number of things:
- Frontend development - I am trying to improve both how we execute but also how Google values it. Frontend development is not always seen as a ‘hard engineering problem’ at Google. I mean we work on a lot of hard things at Google so I get it, but I have spent a lot of time talking to managers and writing supporting guides for performance reviews to help Frontend engineers talk about their work in a way that highlights the difficulty.
- Engineering leadership - I run classes and internal conferences to help grow a new generation of technical leaders, hopefully helping them avoid some painful traps. This is probably the advocacy I’m most proud of. I learn so much about my own leadership skills when I teach and facilitate leadership classes. It’s made me appreciate how little our industry seems to be invested in growing people effectively.
- Engineering Excellence - I am Google’s Code Health Practices lead so I write, edit, and curate a set of documentation on how to improve the health of large, legacy code bases.
Most recently I advocated for changes to a couple engineering job ladders to make sure Code Health was explicitly called out as a key skill for Staff+ roles. I think the omission was an oversight, but it’s not trivial to advocate for changes to a job ladder that affects 10s of thousands of people.
How do you keep in touch with how things really work as you spend less time on hands-on development?
I stay curious. It’s a simple idea, but hard to practice. What it means day-to-day is that I read a lot of docs about all kinds of things happening at Google. I talk to a lot of engineers I work with. I try to still do code reviews from time-to-time. I lurk on 100+ mailing lists to hear what people are complaining about. I just try to really listen to what is going on around me and occasionally fall down a rabbit hole learning about some part of Google I never knew existed. I’m also starting to experiment with pairing office hours where I block an hour or two a week to just sit with engineers on my team while they write code.
How have you sponsored other engineers? Is sponsoring other engineers an important aspect of your role?
I make an explicit point to be a sponsor for all the engineers on my team. I try to coach them through skills growth, advocate for them to our leadership, and when it comes time to promotion I’ll work with everyone of them to help get their promotion packages in good shape. In addition, I try to always have at least one or two folks outside my team that I am working with as a mentor. Investing in making people around me successful is probably one of the most rewarding things I do. It turns out that people really remember you when you invest in them and most are eager to pay you back when they can.
You first got the title Staff engineer at your current company. What was the process of getting promoted to Staff?
Mechanically, it’s really no different than getting promoted to any other level at Google. You may have read that Google’s promotion process is quite involved. There are “360 reviews” + independent committee assessments of the work you do. This historically has involved engineers writing something that looks like a thesis defense justifying their work. The process has undergone some substantial changes lately so this answer may be out of date by the time you read this :). If you are thinking about applying at Google, don’t let this part scare you off!
If I think a little more deeply about the question, there is actually a step before promotion is possible, and that is identifying a Staff-level opportunity. At Google a promotion is typically given once you demonstrate that you can do the work of the next level sustainably. In practice that means that you need to deliver Staff caliber work for something like a year prior to going up for promotion. Depending on what part of the company you are in this can be easier or harder. Staff problems at Google usually involve setting the technical direction for multiple small teams (3-4 people) or one large team (15+). The projects will typically have time horizons of at least a year. Some teams don’t have many Staff sized problems and if those problems are already being worked on by other Staff engineers you may need to go find another project. Once you find the right opportunity then you just need to do the job and deliver. Googlers like to say that their promotion process is ‘eventually consistent’ and once you are successful on Staff-scale project it’s usually just a matter of time before promotion happens.
What two or three factors were most important in you reaching Staff? How have the companies you joined, your location, or your education impacted your path?
Being at Google I have been incredibly lucky to be in proximity to some enormously talented engineers. Watching them work through tough decisions or deal with various kinds of crises has helped me grow much faster than I might have at another company. To be clear, I don’t think this phenomenon is unique to Google. I think anytime you can work with people who have a lot more experience than you, but who also care enough to mentor you, it’s really a perfect training ground. This can happen at large companies or small ones. The key is that you are in an environment that supports your development and puts you in proximity to real world experience that stretches you.
The important thing I had to keep in mind, and still do to some degree, is that growth is often uncomfortable. Learning new skills, trying techniques, working in new ways - all of it requires vulnerability and taking the risk of being wrong. Getting things wrong is a matter of when, not if. As I approached Staff I found myself having to grow a lot in ways that I wasn’t really prepared for: greater degrees of delegation, communicating with directors and VPs, less time writing code, more accountability. I had to learn to keep track of my mental health more closely. I’ve since taken up therapy and mountain biking, both of which help me manage the discomfort I feel as I learn how to navigate bigger and more challenging problems at work (and also in life).
The one other factor I think I can credit for my ability to rise to Staff Engineer is that I really invested in learning how to communicate, both by writing and by speaking. I’m convinced that being a compelling and clear communicator is a powerful skill that can increase the impact of the work you do. Whether it is making the case to fund a project, or changing the morale of the team. Being able to communicate with confidence to a room full of people, or write detailed and useful design docs, can be the difference between breaking through tough problems or remaining stuck. At the Staff level getting stuff done almost always means working with other teams of people. You need to learn how to communicate well to be effective.
I know that communication is a tough skill to learn. Introverted folks don’t always feel comfortable putting themselves out there. Mistakes made in communication can amplify feelings of self-doubt. Not to mention that the way we define successful communication is often based on what has worked for white men in power.
Having said all that, I would encourage every engineer who wants to see more success - even if that doesn’t mean becoming a Staff Engineer - to work on improving your communication skills. Start a blog, teach a class, do something to practice connecting and communicating with other people. Find your voice and then use it.
There is a popular idea that becoming a Staff engineer requires completing a “Staff Project.” Did you have a Staff Project, and if so what was it?
I did indeed have a Staff project because at Google it is practically a requirement. As I mentioned in the promotion process question above, you have to demonstrate that you can do the job of a Staff Engineer for at least a year before you will get promoted. My Staff project involved building and running two subsystems in a larger customer support voice solution. I was responsible for the ‘soft phone’ application that ran on customer support rep desks and I was also responsible for the high performance configuration data storage layer for the entire system of 15 plus services (you’d be surprised how complex a customer support business can be!). I was the TL on the softphone application for a number of years (it actually got me promoted to L5 and L6). In order to take on the new config storage system I needed to delegate some work and I decided to train a new TL. This allowed me to focus more time on bootstrapping the new system. Eventually, I put a TL in charge of the config system as well and I acted as “TL of TLs” for both systems. Juggling between the two appears to be enough to have gotten me to Staff.
In hindsight, I don’t think I would recommend this exact approach to others. The projects were just too different from each other and that made it really hard for me to context shift between them. I think if you want to run multiple projects at once, it’s a good idea to try and make sure there is some “synergy” between them so that you don’t have to maintain two distinct sets of context.
Can you remember any piece of advice on reaching Staff that was particularly helpful for you?
“What got you here, won’t get you there.” This was something that I heard a lot when I was a Senior Software Engineer who was curious about being a Staff Engineer. At first it seemed like one of those management tautologies. Over time I came to see it more as a challenge. What am I doing today? How am I doing it? Is this what I need to be doing to continue to grow? What would it look like if I did things differently? When it came to the Staff promotion specifically, this piece of advice was spot on. As I mentioned above, the skill set needed to be a successful Staff Engineer at Google looks very different than the levels before. If I hadn’t challenged myself to approach leadership, technical decision making, or planning differently, I don’t think I would have been able to get to Staff.
What about a piece of advice for someone who has just started as a Staff engineer?
Don’t panic! You have arrived at a place in your career where you are going to need to grow entirely new kinds of skills. It is going to hurt a little. You are going to make mistakes. It will feel a little like you are learning to do things for the first time. THAT’S OK. Don’t get too in your head about it. People clearly trust you to get the job done, and like you have in your career before, you will figure it out. All you have to do is be a little patient and open to the idea that there are new ways to solve problems. Not everything is a technology problem. Some of the hardest and most rewarding solutions come from getting people around you to work together to build something bigger and better any of you could have built alone. YOU CAN DO THIS.
Did you ever consider engineering management, and if so how did you decide to pursue the staff engineer path?
I actually manage right now, but it is a very small team. Most managers at my level are easily managing 30+ people. I’m managing around 15, but only two people directly. I have found I actually enjoy management of a small group and as a Tech Lead/Manager I am able to join the people development brain with the technical execution brain and it saves me a few negotiations with another manager. Having said that, I don’t necessarily recommend this route for everyone. Being a manager is hard work and requires a whole different set of skills to being a Tech Lead. Worse, if you do a bad job with management you risk hurting people’s lives a lot more.
I do think management is something more people should try in a controlled fashion, but make sure that you are doing it for the right reasons. Don’t become a manager just to get promoted. If you do it, put your whole heart into it because your people are counting on you.
What are some resources (books, blogs, people, etc) you’ve learned from? Who are your role models in the field?
General People:
- Martin Fowler - A legend in this industry. So much of my early career was influenced by his thinking.
- danah boyd - such great insights into the (not always great) impact of technology on people.
- Brett Victor - I sometimes feel like he has been given a gift from the future and we are just finding out about it now.
- Randall Monroe - He has such a gift for making the (highly technical) absurdities of our world so easy to laugh at.
- Neal Ford (from thoughtworks) - Truly excellent communicator. He inspired me to learn how to give a better tech talk.
- Venkat Subramaniam (from Agile Developer) - Similar to Neal, Venkat has such energy and a passion for his craft.
Books:
- The Design of Everyday Things
- Clean Code
- Growing Object Oriented Software Guided by Tests
- The Pragmatic Programmer
- Working Effectively with Legacy Code
- Refactoring
- Object Thinking
- Godel, Escher, Bach - This one can be controversial, but it really got me to stretch the way I think about computing and that in itself is worth a lot.
Blogs/Podcasts:
- High Scalability
- InfoQ (used to have a ton of great videos from so many conferences)
- infrequently.org
- Stratechery
- apenwarr.ca
- Thoughtworks Podcast
Googlers who’ve inspired me:
- Robert Konigsberg - Who helped me truly find my voice.
- Titus Winters - Who has given me so much support and opportunity over the years.
- George Fairbanks - Whose deep thoughts on the craft of software development have caused me to think much more deeply about the work I do.
- Michelle Levesque - Who probably doesn’t remember me, but she showed me what authentic and inspired technical leadership can look like.