They say, when you have lemons make lemonade. So I did.
A little warning, it is going to be a long blog post…
In another post, I described how I go about plans. I make lists and check items as I get them done. I approach things with structure. I like structures and routines. Not least through my time at TU Delft, where I was looking at human behavior, I started to think more about habits, and routinized behavior. I like habits and routines. This reduced the number of decisions I have to make and thus make my day-to-day activities more efficient. There is such a thing as decision fatigue. Habits help us to lead successful lives. For example, in the book Atomic Habits (Clear, 2018), one learns that it is through habits that we can stick to healthy (and unhealthy) behaviors. Apparently, successful people reduce the number of decisions they have to make and instead create habits and environments that support these habits. If you, by default, go to the gym every second day and build structures around it, it is likely you will go to the gym. If you start an inner monologue in which you negotiate with yourself whether you go or not, you have already lost. So, it is habits and the structures around them that help us stick to good (and bad) behaviors in our lives.
But for sure, habits and structures can get us locked in. We get rigid and not able to change, which eventually will lead to a breakdown. That is because life changes and at times, it is wise to adapt. Soon I will travel to Australia for a conference. As much as I like traveling, I know it disrupts my habits, and that makes me a bit anxious. If I would be completely unable to accept change, I would not travel and lose out on certain life experiences. Thus, clearly sticking to structures and habits at all times is not advisable. I guess one has to find a healthy balance.
I know that when I am in Australia, I will perceive everything through my filter. I will apply my cultural background, the things I have learned, seen, and experienced so far, and compare it with that to make sense of these new experiences. I might compare the vastness of the country to my home country, which you can cross by car in one day. I will compare the food, the animals, plants, roads, houses, etc, to what I am used to. I will behave as I am used to, and I might feel confused if people act in ways I am not used to.
When we travel, we expect to experience new things, and we expect that we might have to adapt. But what if you get into a situation in which you do not anticipate this? Recently, I got confronted with such a situation, and I had to learn to adapt and make lemonade. My mind is an interesting thing. On the one hand, I love my structures. On the other hand, I think I am a very creative person. There might very well be a relationship between structure and creativity. As we will see, the post will expand on this topic (If you think my posts are too long, how about listening to this nice podcast about creativity hosted by Professor Humberman). In any case, it is my creativity and openness that helped me to apply a new frame to an unusual situation. To shift gears, restart, and see what I can learn. And it is my love for structures that allow me to see patterns in the seemingly chaotic.
So, what on earth is this post about… It is about Agile in academic research. No worry, if you do not know what Agile means, I will explain it in a bit…
When I do not see a way forward, I look for help. That includes talking to experts, mentors, friends, family, and peers (BIG THANK YOU to everyone, I would be lost without you), as well as reading or watching YouTube videos . By accident, I stumbled upon Agile and started reading. After a bit, I figured that this could be a useful frame, and I think it is not only for me, it could be for many. Hidalgo (2019) did one of the few studies about Agile in academia. To study different aspects of Agile, he interviewed researchers who use Agile principles. One of the interviewees stated:
“We were already working with an agile approach but we had not called it ‘agile’. Later on we started formalizing things and picked up more and more scrum tools and techniques to improve the way we manage our projects.”
From this quote, I take that my experience is not unique. Furthermore, reading literature about sustainability (transition) projects, I came across statements such as:
„The structure of any sustainability science project should be conceptualized not as static, but as a dynamic process with a clear vision but no final state defined in detail. This combination of clear objectives and flexibility in structure and content should be backed by a number of contingency rules […]“ (Spangenberg, 2011, p. 282).
You may not know what Agile is, well, I did neither. Thus, I had to find more information. There is not a lot of scientific literature about Agile in academia. Heads up! This is a research gap you can study… The few scientific papers, I could find, did not quite provide me with sufficient information, and the new vocabulary was confusing (SCRUM, Kanban, sprints, backlogs…). Thus, I turned to books. I picked two books randomly that seemed to cover the matter. Later I bought a third one that is actually about Agile in academia. As stated, there is not much about applying Agile in academic settings. I appreciate reading the book The Agile Faculty by Pope-Ruark (2017). Though I think she is a bit overly enthusiastic about Agile. She applies Agile to the US university context. Arguably, universities in the USA differ quite a bit from universities, say in Austria, where studying is mostly free. However, her book might be a good source for those who want to apply Agile in academia. In any case, we need more literature like this, because I think one cannot just apply Agile 1-to-1 in academic settings. Furthermore, there is quite some literature about the downsides of Agile in business settings, and I think academia can learn from these lessons.
What is Agile
Agile is more than a project management tool. It is about processes, structures, culture, people, and leadership. So there is much to unpack. Too much, in fact. I am mostly focusing on Agile processes. But, of course, one cannot really separate processes from the other aspects. E.g. when you want to apply agile processes but the organizational culture is not aligned, you might not be successful. Think back to my beloved habits and the structures you put in place to make them happen. It is not different for Agile processes. When you want to work Agile, you have to set up Agile structures.
The process
The Agile process is coined by feedbacks. Instead of thoroughly planning your project beforehand, you determine a vision. The vision forms your overall goal that you want to reach. It is your lighthouse and anchor. How you reach the vision is open and can change along the process. After you have determined your vision, you formulate smaller objects (in Agile, these are formulated in a specific way and called Epics). These objects are then further broken down until you get workable junks (these are called stories and tasks, where tasks are the smallest unit). Basically, you write a story and break the story down into smaller stories, so you arrive at actionable steps. Think of it as the vision being the marathon, and you split it into smaller sprints.
The stories are collected in the so-called backlog. Then you make a priority list within the backlog. Based on the priority list, you select the first task(s) and determine a timeframe for the completion of the task. In Agile, the timeframe is between one to four weeks, and this timeframe is called a sprint. Depending on your circumstances, the sprint can be longer. But I think the essential is that you break objectives down into realistic, actionable tasks and to commit.
In essence, you build a comprehensive to-do list and create structures to accomplish your tasks. Now the interesting thing about Agile is that after the sprint, you reflect on you how things are going. Did you manage to do your task? If not, what is the problem? Can you do something about the problem? Do you have to shift priorities? Do you have to get something else done first, etc.. Did it work? Why? What are the success factors? What have you learned? I think it is very important to highlight that the revisions are not normative. It is taking stock for the sake of learning. You learn why things work and why they do not. Both is important!
To explain, I will use a sports example. I am into handstands, and sometimes I observe people trying to learn how to handstand. They repeat their approach over and over and over and over again. They seem not to reflect on their approach, and they might not ask for help (reflection with an external). So they might not progress and do not find out why. Potentially at some point, they will give up, although they might have been close to achieving their goal. If they get some hints on what to do, they might suddenly make progress. Others may manage to get in the handstand, but they do not understand how. If they are asked how to do this, they might not be able to give a good answer. So if you do not reflect, you cannot replicate your successes, and you cannot learn from your failure.
How does that relate to the feedback and Agile. If you want to learn the handstand and start by kicking up, you should, check in and see how your training approach works. If you find you are not successful, you reflect, and you figure that you lack shoulder and core strength. Then it makes sense to adapt your training and first work on other aspects before your revisit the kick-up. Or you integrate certain strength exercises while you work on the kick-up. You make a new plan, commit, and after, say, three weeks, you check in and see if it works or not.
It would be stupid to set up a training pan for a whole year and not deviate if you realize it is not working. Yet, traditional project management often works like this. Agile, in contrast, sets the vision (learning a handstand) and allows the path to get there to be flexible.
For the research context, it is similarly argued that it might not make sense to set up a detailed plan that you may not be able to follow and then spend time legitimizing why you had to deviate from the plan, although it was clear from the start that you will deviate from it at some point. Thus, the argument is that one loses a lot of time, working like this. Furthermore, it is argued that one gets stuck in a rigid plan and that the results might not be optimal, because one tries to press a plan onto circumstances it does not fit.
Structures
To allow Agile processes to work, you need to set up specific structures. One can distinguish between the structures within the respective project and the structures the project is embedded in. The latter would be, the university or the research institute, as well as, the structures of academia as a whole. More on that later. First, I expand on the structures within the project.
Agile projects should be supported by structures that allow learning, reflection, creativity, and innovation. The ideal setting are small multi- or interdisciplinary teams (7 +/-2 people), working in parallel on different aspects of a problem. What part each team is working on has been democratically agreed on. Above I have described the process. Obviously, the teams are involved in defining the objectives and tasks, as well as ranking them according to priority. Also, teams commit to these tasks, rather than tasks being assigned to them. That creates ownership over a task which should have a positive influence on people’s engagement. Furthermore, the teams are free to find their own ways of solving the tasks, they have committed to. Hence, teams are empowered to be active, creative participants in the process.
The teams should be multi- or interdisciplinary to facilitate out-of-the-box thinking and to allow end-to-end solutions. For example, design decisions of a tool might need to be connected to insights from behavioral science and material science. Without wanting to expand too much on an aspect I have been completely quiet about so far, but which is actually core to Agile, I have to mention that the solution should start with the stakeholder (customer) needs. The stakeholder orientation of Agile is worth another blog post, and I have to omit it for now.
In Agile projects, hierarchies are flat, but there is still someone who is in charge of keeping an overview. It is clear that Agile supports leadership rather than top-down management. Nevertheless, Agile does not mean that the overall project lead can have a hands-off approach. Personally, I think that a leader needs to be much more skillful than a manager. I won’t expand on the people and leadership aspect in the blog post. However, it is important to note that Agile leaders need to have certain qualifications and character traits.
Culture
Agile processes do not only need the right structures but also need to be supported by the right culture. Agile is about learning and reflecting. Thus, if people do not feel safe, Agile will not work. It is vital that agile processes are embedded within a culture of trust, transparency, openness, and respect.
The aspect of culture is, of course, connected to the overall structures one is operating in. So I want to go back to structures but discuss them in a broader sense.
Embedding Structures
An important part of agility is to motivate employees. Favorably one should build on people’s intrinsic motivation. Within academia, incentive structures, are a hot topic. Of course, we all want to generate knowledge, learn and solve societal problems. But to do this, we need to stay employed and favorably the employment situation should not be precarious. The entry ticket to a long-term position is, amongst others, publications.
Should we switch to other incentive structures than scientific publications? Have a look at Figure 1. Academia is a battlefield. There are only a few permanent positions even fewer professor positions. To get there, you have to compete and tick boxes. These boxes feed the incentive system researchers follow. Amongst others, these are publications. Agile supports stakeholder-driven (aka co-creation), interdisciplinary research, and disruptive innovation. As Kuhn (2012) describes breaking with scientific paradigms is not easy, and it might conflict greatly with getting lots of publications (you may also want to look at Park, Leahey, & Funk, 2023). It is also no secret that scientists still struggle to get their results published if they perform interdisciplinary research. Stakeholder-oriented research might also not necessarily support peer-reviewed articles but rather communication of results that is relevant, accessible, and easy to understand by stakeholders. Further, stakeholders may not want to wait between 2 and 4 years until the paper finally got through the review process. Thus, this process is not agile-friendly. Not that I do away with scientific publications, but the question is, what kind of culture do they create and support?
Universities are less independent in setting their own incentive structures. For example, in the Netherlands there is a shift to reduce the usage of publication scores and to include other parameters as well. While this is a great shift, an international researcher might care little about this. As soon as I apply in another country, I have to comply with the dominant metrics. That essentially means that two incentive streams need to be created without increasing the burden on researchers.
Organizational structures may also not support interdisciplinary settings and disruptive research. Usually, universities are structured following disciplinary confinements. This does not support interdisciplinary work. Following Perkin (2020), businesses have also been structured in a siloed manner, which did not support collaboration among people with different backgrounds. For an Agile transition, Perkin (2020) describes how companies have been restructured to better suit the new way of working. I am not sure if such a complete overhaul is possible for universities. But it would be an interesting experiment on small scales. E.g. researchers would organize based on problems rather than disciplines. A research team around circular economy would form that includes engineers, all sorts of social scientists, humanities, and natural sciences. Similar setups can already be found in interdisciplinary projects, though these people usually work in different institutes, and thus, they are spatially isolated. Arguably a new sense of community would be created if one would not look at the social scientists over there and the natural scientists over there, but all being part of one team located on the same floor. Such an organization would require the teams to remain agile. Hypothetically, at some point, the circular economy will no longer be a worthwhile topic, and the team has to find a new one. Thus, a new team needs to be formed.
Having said that it needs to be highlighted that Agile should probably not be the one fits all solution. Perkin (2020) highlights that Agile may not be applied at all the time. That is, for example, because innovations follow certain stages. In the beginning- to mid-stage, agile is suitable, in the end-stage, it is not, or not as much. Furthermore, companies, need to be ambidextrous: which means that all innovation horizons need to be run in parallel. To simplify, one could distinguish between incremental and radical or disruptive innovation. Running both streams in parallel provides companies with stability while opening the possibility to venture out into completely new avenues. That is advisable as new developments could make the stable path obsolete out of a sudden.
Arguably, the same applies to universities. On the one hand, one wants to continue with puzzle-solving activities, as Kuhn (2012) would describe it. On the other hand, one wants to work on paradigm-transcending activities. These two paths need different management approaches. Agility fits better to disruptive and less to incremental innovation. In practice, that would mean that universities have traditional incremental areas and disruptive innovation areas at the same time, but managed differently.
In terms of hierarchies, one can also wonder what the adoption of more Agile principles would mean. There might be different views on this, but I think some hierarchies are necessary for an organization. Nevertheless, on a smaller scale, say a research group, flatter structures could be adopted. An observation I recently made, that I think can be connected to culture and hierarchies is the way a group is presented. I might be completely wrong with this, but if your research group is depicted in the hallway, how is this done? Is there a group photo? If so, where are the leaders/managers? Or are there individual images? If so, how are they organized (see Figure 2)? Have a look at the image and think about the feeling it gives.
Some downsides
The time argument
It is argued that Agile is more time efficient because you do not spend a lot of time setting up a detailed plan, which you, in the end, cannot follow. It is also argued that Agile meetings can be shorter and that leaders do not need to spend that much time on micro-managing.
I agree that it is ridiculous to spend time planning if you cannot follow the plan. Thus, yes, there needs to be more flexibility in traditional linear (or waterfall i.e. a beautiful Gantt chart) approaches that can be facilitated by introducing more Agile principles. Though, I disagree that Agile processes do not take time. I assume that at the end of the day, the time invested is equal. Before and after each sprint, you not only have to reflect on what you did but also think about what to do next. That takes a lot of time. These reflections happen on a daily and weekly/monthly basis.
Anyone who participated in democratic processes, also knows that this takes time. It is much easier to just make a top-down decision. So buckle up for some lengthy discussions, where you talk in circles. Thus, I absolutely disagree with the time argument. It can also cost quite some nerves if you do one thing and in the next meeting, it is democratically decided that what you did is no longer useful because the project changes direction. This is not at all saving time. Annosi, Hemphälä, and Brunetta (2018) for example show that product innovation decreases in Agile processes, while process innovation increases. That reduced product innovation relates to the rather inefficient communication structures that Agile process can bring along, if not managed properly.
I also disagree that a leader does not have to micromanage. In the first place, also in a waterfall approach the leader does not have to micromanage. It is 1. a question of good leadership, 2. a question of personal capabilities of the person(s) who implement tasks. Any (good) leader knows that some people need more guidance (and sometimes micro-management) than others. Just as some children need more support than others. A leader who is a control freak will micromanage irrespective of the management process used. Ideally also in a waterfall approach, you have your team working with autonomy. This is why you set up a plan; to allow people to execute it without holding their hand.
In contrast in an Agile setting, much more check-ins with the leadership are needed. You constantly have to check whether what you do serves the vision. If you have an incapable leader in place, that can go very bad. E.g. if a leader is easily carried away or erratic, you may do a lot and nothing at the same time. That ties into discussions about leadership qualities and matching process management approaches. Potentially a leader who has a tendency of being erratic might better apply a waterfall approach because it helps that person to stay on track. This is not to point fingers, but a self-reflection. I am a very creative person. If I do not keep this in check, I get nowhere.
So if you think you want to apply Agile to save time and nerves, think twice.
Toxic Agile
I want to highlight that Agile does not mean less workload or that you can reduce stress just by applying Agile. Pope-Ruark (2017) implies this somewhat in her introductory chapter. Agile is presented as THE solution. However, as Rigby, Elk, and Berez (2020) highlight, Agile can be abused to increase work pressure. Thus, Agile is for sure not a silver bullet.
Pope-Ruark (2017) highlights the values and principles of Agile and Scrum (a specific tool to facilitate Agile) and implies that these contain better values than those of waterfall processes. For example, Agile emphasizes trust, openness, self-empowerment, or autonomy. These are great values, but it is not true that these could not be applied to other project management methods. Building trust is relevant in any context! That employees have the feeling of ownership, and of making a real contribution is always relevant. On the flip side, some of these values could be abused to pressure employees. For example, self-empowerment could lead the project leader to put the responsibility on other people. Or it could lead to situations where no one is responsible (Rigby et al., 2020). Toxic work environments can be created no matter which process management approach and methods are used.
Some of the Agile statements have a toxic aftertaste. Pope-Ruark (2017) states that in an Agile Faculty, you should ask yourself, “how committed do you feel to your goal?” Thus, it is not about your progress but your commitment to the goal. I read this and see a motivational speaker on a stage getting everyone hyped up. Don’t get me wrong, I am passionate about what I do (too passionate, in fact), but addressing deep incentive structures can be dangerous. Pope-Ruark (2017) highlights that it is not about productivity but about our mindset, about our vision, goals, and commitment. That can be abused in many ways. Managers can, if we do not perform properly, now question our commitment. I want to say that because agile emphasizes deeper psychological incentive structures, it can also be abused.
“Because Scrum focuses on supportive collaborations, clear understanding of what needs to be done at any given time, and positive professional values, it is an excellent match for faculty work, especially when we seek to pursue professional vitality, increase engagement, and think with a growth mindset.” (Pope-Ruark, 2017).
If you do not perform well, your manager can criticize your mindset or values. I think that this has dangerous implications.
Some closing thoughts
I think the real strength of Agile lies in the emphasis on circular processes. However, many of the values that are ascribed to Agile should be values applied in any process management context. Thus, I do not think that they are THE specific strength of Agile. But what is specific is the recognition of circularity, and thus its emphasis on learning, creativity, and reflection.
Agile also focuses on people. People are understood as THE asset of a project. It is people who are creative and birth new things. I think that traditional waterfall approaches could also be people-centric. For example, you should anyway focus on your stakeholders if you call your project co-creative or transdisciplinary. But where in a Gantt chart do you integrate the freedom to change the path if you learn that a certain path might be better than the path you initially envisaged? I think reports often present a process as a logical sequence of events. But they miss out on explaining that, in fact, while you were in it, it was a mess.
Agile supports creativity, learning, reflection, and innovation by providing very specific structures and a very specific culture. Agile provides different structures than waterfall approaches, but they are structures, nevertheless. They require stellar leaders that are excellent at dealing with people and complex, messy situations. You need to have the ability to keep an eye on the price while floating around in chaos. You need to be able to handle democratic processes and thus be in deep respect of different points of view. You need to understand the needs and priorities of everyone involved and try to acoomocate them. Why are we using Gantt charts and detailed plans? Why do we think we can plan transitions? Because humans don’t like risk. As I understand it, Agile is not for the risk-averse.
References:
Annosi, M. C., Hemphälä, J., & Brunetta, F. (2018). Investigating the Impact of Agile Methods on Learning and Innovation. In P. Boccardelli, M. C. Annosi, F. Brunetta, & M. Magnusson (Eds.), Learning and Innovation in Hybrid Organizations: Strategic and Organizational Insights (pp. 73-97). Cham: Springer International Publishing.
Clear, J. (2018). Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones: Random House.
Hidalgo, E. S. (2019). Adapting the scrum framework for agile project management in science: case study of a distributed research initiative. Heliyon, 5(3), e01447. doi:https://doi.org/10.1016/j.heliyon.2019.e01447
Park, M., Leahey, E., & Funk, R. J. (2023). Papers and patents are becoming less disruptive over time. Nature, 613(7942), 138-144. doi:10.1038/s41586-022-05543-x
Perkin, N. (2020). Agile Transformation. Structures, Processes and Mindsets for the Digital Age. London, New York: Kogan Page.
Pope-Ruark, R. (2017). Agile Faculty: Practical Strategies for Managing Research, Service, and Teaching: University of Chicago Press.
Rigby, D., Elk, S., & Berez, S. (2020). Doing Agile Right: Transformation without Chaos. Boston: Harvard Business Review Press.
Sovacool, B. K. (2023). Reprint of: The privilege of learning and serendipity: My principles of publishing research for a new academic era. Energy Research & Social Science, 100, 103133. doi:https://doi.org/10.1016/j.erss.2023.103133
Spangenberg, J. H. (2011). Sustainability science: a review, an analysis and some empirical lessons. Environmental Conservation, 38(3), 275-287. doi:10.1017/S0376892911000270