Getting through a big project

@kjuulh

2023-12-09

Introduction

As a engineer you will work on a variety of projects, some small, some large, some complex, and some you can handle in 15 minutes. But once in a while you will be faced with a big project. You may not know initially that it is gonna be a big project, but somehow it turned out to be.

In this article I am gonna share some of the experience I've accumulated, working on big projects, the various stages, and surprises you may will face, and that it is gonna be okay.

This article is mainly geared at new software engineers, but is applicable to most engineers as it focuses mostly on the problem domain, not the technical skills.

What is a big project

I categorize a big project to be around 3 months or longer. This depends on your own internal barometer of what you can stomach but to me, I like to do small changes, fast, iterate and so on. 3 months and longer would be a project me and the team would have to work on for a long time, and face a natural project development lifecycle which I will share later.

A big project are usually constitute a few factors, each factor you add to the project it will multiply the amount of effort required.

  1. Cross team collaboration:
    1. Do you need external parties to succeed.
    2. Do you need to break some interface to another team, i.e. does a design change, which requires work from either 1 or more teams?
  2. Complexity:
    1. Does your team have the skills to solve the problem at hand. This may be rooted in how new the team is, if it is new, add a multiplicative factor. If it is highly technical, that is outside your immediate skillset add a multiplicate factor
  3. External domain:
    1. Are your team developing this feature on behalf of another domain or product?
  4. Team size:
    1. Do you either have a too large team, or a too small one. This may seem counterintuitive, that is that in some cases it is preferable to have a small team to be agile, but for other kinds of projects, getting a good culture and collaboration can take longer than actually developing the project.

I'd go as far as If you'd estimate a timeline initially for the project you should multiply it with each of the above.

4 weeks * 1.5 ^ (1 + 1 external parties + 1 external domain + 1 small team)
= 4 * 1.5 ^ 4
= 4 * 1.5 * 1.5 * 1.5 * 1.5
~= 20

The above may seem extremely, but if you need to involve that may parties there is a high likelihood of anything splitting the time, a simple miscommunication, etc. The above is just a naive example, but I've seen big projects exceeding timelines by years, because of complexity of collaboration or purely technical reasons.

In my experience you don't get a big project unless you've got at least one of the above, I will dive more in depth into each, but think of it like this; a project has multiple stages of development each of the above will add roadbumps at each level, and to learn how to navigate these requires experience, leadership and persistence.

Why big projects can feel like pulling teeth out

I don't think I've ever meet a person whom likes these long drawn out projects, fraught with extended deadlines, problems, conflicts and whatnot. In this section I will dig into my experience of why this happens, and tied to the previous section, on how to navigate this space, and keep a cool head.

Like many things, in acquiring knowledge about a subject, it is not a straight sin curve, it is a journey filled with entropy and seemingly random events, which are frustrating and causes problems.

Like the Dunning-Kruger effect it follows something like this

  • Stage: 1: the project has been pitched
    • At this point you feel: Optimism, confidence and hubris; You've got this, you and the team are just beginning to collect requirements (hopefully), and everything looks good, it may seem a little overwhelming, but you feel like it is gonna be great, you've learned from the past projects and this is gonna be a breeze.
  • Stage 2: The requirements just keep coming in, and nobody wants to sign off on what the scope is:
    • At this point you are still in good spirits, you've maybe just finished the first part of your PoC, and have a plan to solve the initial requirements. You and the team are beginning to get frustrated with the ever expanding requirements, and meetings. But you're still confident you can solve the problem within a reasonable amount of time
  • Stage 3: The requirements just don't stop, meeting with stakeholders, doesn't go as expected, major delays from third party vendors are happening, sister teams have shifting priorities, it is impossible to get sign off on the project and your solution is half baked at best. You also lack key personel to finish the job in time:
    • At this point you feel burn out, exhaustion, and an assurance that you're not go finish in time, however at this point you communicate that you can't make the scope, and with the stakeholders you have a crisis meeting where finally, you decide on a minimal set of features required.
  • Stage 4: the project is still half baked, and you continue to face problems, but you have confidence that the problem can be solved, you have pretty much full domain knowledge at this point, but the sheer amount of work required is daunting. Now you just want to get over with it.
  • Stage: 5-: At this point either you round off the project, or you continue in smaller iterations of the previous stages, it depends on how long the project is and what the scope was decided to be.
    • At this point you will discover flaws in your solution, but no allocated time to fix it. You may have decided with your project owner to hack something together to finish the project in the earlier steps to finish the project in time. But now you don't have time to go and fix it. The cycle either continues with another project, or the same project in another iterations.

The above bullets are a bit extreme, but I've faced variants of these in nearly all of the bigs projects I've seen. Some have been easier to deal with than others, but all in all, experience is what counts. It is also why some engineers seemingly have a higher titles than others, you may feel like your technical skills are on par. But what counts is that you help navigate these projects on behalf of your team, shield them, guide them and not least mentor them. (Thanks; Anders Keller Carstensen, Martin Anker Have and Bjørn Hald Sørensen)

It is always easiest to see these projects in hindsights, but the most valuable thing is to identify when you are about to enter one of these, so that you can come off to the best start to them as humanly possible. You will not be perfect, but doing the best work you can and take the conflicts required, will help mitigate a potential disaster.

Experiences

In the next section I will dive into how to mitigate each of the above, it will not be a complete list, as my experience isn't complete. But it should still be useful, it also won't be bullet point by bullet point, but rather some specific topics which should help solve key problems.

Requirements, requirements, requirements

Requirements are the absolute most important part of a project, it doesn't matter how much of an experienced or skilled engineer you are, if the requirements aren't clear, you are either toiling away for no avail, or trying to built the tower of babel. It is not gonna happen.

Requirements seem like such a bening thing, but the difficult part of gathering requirements are two fold;

  1. You don't have complete knowledge
  2. Requirements are decided by a non-technical partner

You don't know everything initially and discussing solutions with a non-technical partner is a learned skill, that is difficult to acquire, you've got to be confident in your teams solution, and methodology, but also being steadfast and be an effective communicator with a non-technical partner.

My best advice here, is to exist as a unit, you may decide to be the person to gather the requirements, gather the responsible people in a room or meeting, poke for the requirements over several interactions with a clear agenda. Get consensus, and clarify twice or thrice. And if you're being pressed to agree on a solution, use we, say you need to discuss it with the team, and get back to the steering committee, for the project.

An example could be:

Business person A: We need to have National login implemented within 2 months, otherwise people cannot login to our platform

You: It will potentially require a lot of work, I will need to discuss with the team, to tell whether it is possible or not

In the past, I would try to estimate this on my own, and give an immediate answer. Don't. Whatever you give here, will set the wrong precedence. Work as a team, and rely on them. If you are the sole technical lead, discuss the best course of action with your team, manager. There is no silver bullet here, but for software the best approach is to say that it is possible within a given criteria, you may need extra hands, extra focus, or simply that it isn't possible.

We are engineers not magicians. Don't present a concrete answer to a question you don't have an actual answer for yet. It always ends in grief.

The interface is everything

If you need ongoing consensus between teams or external parties you are doomed from the start. If you need to continually interact with a second/third party to develop your project, then you will move at a snails pace, and won't be able to deliver on time.

The best strategy is:

  • Sit together
  • Have tech leads collaborate continually
  • Scope the project differently

If you can't do either of the above the project simply isn't possible or sound.

The first is ideal, if you need a delivery to another team, actually transferring that domain knowledge to them directly by peering is the best. Usually, because of business politics and bureaucracy it isn't possible, sadly.

The second is a stop gap, and will take away resources from the team, this only works if the techlead isn't the only person on the team with the technical experience to drive a project. It requires a lot of work from said person, but can make the project succeeded. It is usually done by having the two teams senior technical leaders communicate on requirements day by day.

The third is thinking critically, can you scope the project in a way that another team doesn't need a concrete delivery by the team you're on. This may be deciding on an early interface or design between the teams, and then implementing it afterwards. This usually requires follow up of either of the previous points as well, but on a lesser degree. Requirements will change and you will need to change the design on an ongoing basis, so plan for the communication.

The conflict not taken

As engineers it can be damned difficult to take a crucial conflict. This is both a learned skill and a natural inclination. Some people are just willing to fight tooth and nail for a solution, and others don't even want to look a stakeholder in the eyes.

This is a make or break moment. I don't have this natural inclination for conflict, but have learned to follow through and rely on certain techniques to make it easier to actually remain steadfast and resolve said conflicts in a respectful way.

If you are new, you can be owned by a seasoned business professional, be aware, and use the we technique described above. Sometimes other engineers or business professional can step over the line, if they do so, you should rely on your leadership. This may be either a manager or director. Sometimes you may need that distance and that is okay! especially as you learn to tackle these things.

Why are we doing team meetings all the time

A problem requires a certain set of skills and conditions to solve, and it is likely that your team dosen't fit the bill perfectly, you may either be too small, too large or don't have the technical skills to give a proper evaluation of the project. I define a small team to be 2-3 people, and 5-10+ to be a large team.

A too small team means that you will be impacted greatly by changes to the team, or simply a lack of hands or capability to solve said problem. If this is you, you need to set the correct expectations to your stakeholders that the solution will have an undefined deadline. Let your leadership have the word here. It can be very difficult to communicate this is in a good way.

A too of a large team: funnily enough this can be even more difficult to solve than a too small team. It is not actually as rare as it might seem, but given a large team, that wants to act as a unit. A consensus is often required. It can take a long time to get consensus for even simple ideas, or features. My solution here is to simply to communicate that we shouldn't have consensus for everything. And rebuke the notion that you need a scrum master or agile coach or whatever to drive your teams culture. It is fine to have team events, encouraged even, but don't fall for the company culture events. It is bullshit in my experience, you learn by working together, and not all at once. Work on problems 1 to 1, and solve problems that way. Large discussions can be had, but if you decide to open said discussion be prepared with what you think will be an acceptable answer beforehand and be ready to defend it.

Damn, this project is drawing out

This will never become a solved problem, but you may become more equipped to handle it. A project will have many cycles, over months, years, weeks. It can change at a flip of a coin or be drawn out until you don't feel like you're making any progress anymore.

This point is the simplest and hardest at the same time. This means that you will gain the experience to face these projects in time, but will never find the correct tools to fix them. We're sometimes prone to refine the tools for the project we just solved, but those tools won't be sufficient to handle the next project on the pipeline. Because if they did then it wouldn't be a big project.

With more experience you will to guide your team and yourself through this these phases, you won't be unaffected, but you've been here before. So you now what is going on in the trenches, but you're still in the trenches. It is a constant balance, and the most important is to listen to your team, we each have different capacities, and we have different breaking points. Be aware, that especially Junior developers should be shielded from too much complexity, but also that stress needs to be manageable for everyone.

Deadlines, always deadlines

A deadline is such a frightening thing. It is a target, and guillotine at the same time. But from experience that is not actually true, at least not for reasonable organisation. A deadline is a target, yes, but not a guillotine. After a few projects you will notice some patterns. A project can always be extended given proper cause. It sucks having to say that something cannot be done in time. But it has to be done.

As I always say to the teams I am working on, a project has three levers you can change:

  1. Scope
  2. Resources
  3. Time

If requirements are set in stone then scope cannot change, if you cannot get extra headcount then resources cannot change. As such the only lever you've got is time. Time is your last resort. But it is possible to extend a deadline. Have a talk with your manager / product owner, often it isn't as bad as it may seem

I only see walls

Tunnelvision is a real thing in these projects. When you've worked on them for a long time, you lose your objective sense of how important the project is, how valued it is, how well your solution fits, etc.

There isn't a concrete solution here, other than being aware, that if your project isn't the business' primary goal, then it won't get extended funding, or that the deadline isn't as important as it may seem.

If it is as important as the business say it is, they will throw money at the problem to get it fixed. If not, then it isn't as important as it may seem to you. This can be a bitter pill to swallow, as the project you've toiled away for, may not actually bring the value to others as it initially seemed.

This is also why it is great to work iteratively, but never giving up quality, solve a problem just as good as you can but not better. I've never seen gold-platting giving any benefits in the past, other than having to break it all down and starting from scratch.

We'll do this now, but fix it later

This is a lie we tell ourselves when giving estimates, we'll just do this little hack here, and then get back to it next quarter to fix it. If this is what you tell your business leader, in 90% of cases, your team doesn't have the maturity or agency you think it does.

You won't get back to fix this dirty hack, it will be a stain on your conscience of the next years, and the business won't care. If you can't professionally agree with a solution then don't propose it. If it sets you back 10 steps, and moves you forward 1.5, is that really a win? no.

Communication, I'd rather not

Communication is the most vital part of a big project. You'll have stakeholders, and partners. There is a fine balance between oversharing, and undersharing. You may choose to keep key learnings to yourself, because you know that it is gonna risk the deadline. Or you may overshare causing teams to have collaboration meeting each week, with no direct benefit.

Do not plan regular meetings unless many teams need to succeed on a high level task. Do share your learnings internally and ask for feedback from your manager / product owner. They will need the information to share more broadly, and if you have to do it yourself, know the recipient. It is a learned skill to be an effective communicator. One which, I've struggled with my entire career at, sharing concise information in a language that the recipient can understand and act on us crucial.

I will go into this in a future article for self development in this space. Just know that all communication has to be intentional, and targeted at a specific reader, and you're off to a good start.

I can't do this anymore

It may not be in every project, but especially for newer teams, someone will break. Whether it is stress, not the right expectations, giving up, etc. It is important to listen to yourself, and the team. We don't have the same ideals, work ethic, interests, etc. As such you may feel that the project is bearable, but another may be on the brink. Make sure you communicate, if you're the technical leader, make sure to set the right expectations of your teammates, they are not you. Don't project your skills onto them. You may have a vision for the project, but they might not know or have the skills to implement it.

Doing these projects is much more a team sport than a personal one. You will find that as a technical leader, you will spend more and more time, communicating and mentoring rather than implementing. As such, keep your ear to the ground and learn to mentor instead of ripping away the keyboard from your teammates. They are not you!

If you're a junior, if you see your tech lead being stress. Open the communication as well. They may be bottling up all the technical problems to themselves. They can break just as well as any other.

I've been in either situation and it is just as hard. But I've never had a bad experience asking a person how they felt. And if they rebuke you, then the company is probably not one that values you.

Conclusion

In this article I've shared some of my thoughts on how big projects are being done, and some of the experiences and learnings I've faced while doing them. It is by no means a complete list (and if you feel something is lacking reach out at contact@kjuulh.io, I am also available for general mentoring as well). But it should hopefully give some ideas, and help newer engineers navigate the difficult space that is project development and collaboration