Scrum, by Jeff Sutherland: The Book EVERYONE Should Read (Part 1)

Most people think this book — Scrum: The Art of Doing Twice the Work in Half the Time — is specific to software development. However, that's a very unfortunate misconception. Scrum is about being productive, it mostly doesn't matter the task. It's actually an algorithm of iterative planning — i.e., instead of making long-term plans, we adjust the course periodically —, encompassing not really only teams but also individuals — in fact, I've been using it for a while to organize myself and it works wonderfully.

Scrum, by Jeff Sutherland, is also incredible even if you're very experienced with Scrum. It discusses why Scrum is like it is.

If you need real world examples of successful projects that used Scrum, there is no shortage: from complex software to hardware, cars, changing the whole structure of FBI investigations, managing news teams, improving (revolutionizing) traditional classrooms, changing how governmental projects are developed, etc.

In this post(s) — this will probably be a series of posts actually —, I'll be spoiling the book completely: I'll summarize the major concepts and solutions Jeff Sutherland, the (co-)creator of Scrum, exposes in it. But that should not deter you from reading the book actually, the stories he tells us are amazing by themselves, and they do reinforce the arguments he wants to prove by a lot. As he mentions in the book: the stories provide the why of the methods, while the methods themselves are extremely simple and can be learned in 10-minute videos on Youtube.

If you're interested in practicing Scrum right away, besides the traditional physical board and sticky notes, you could use some quite neat software solutions. The one I would really recommend is from Atlassian: the famous Trello. Another option, if you're building software, is to use the Issue Boards from Gitlab or the Projects Boards from Github.

I would also like to remark that Scrum is not another management fad to make directors and managers look cool. In fact, Scrum is a way of getting bullshit out of the way. As an iterative method, if the team thinks that the introduced "improvement" is not helping at all, they discard it in the next iteration: and that's the quick end of the dreaded fad. Scrum is also partially based on the Toyota Production System (Lean Production Systems), a more than half-century old system idolized by top executives around the globe, and which, curiously, is in turn based on US industrial best practices from the early 20th century. And, if that isn't enough to convince you of its value, one of its founders, Jeff Sutherland, is a Korean War Veteran Pilot, a West Point Engineer, a Stanford Statistician and a PhD in Biometrics by the University of Colorado of Medicine. So, in short, Scrum is based on empirical and scientific methods, there will not be any esoteric bullshit.

Before continuing on to the bulk of this post, I would like to reiterate that the biggest value of the book is within its stories. It is through them that we humans (perhaps unfortunately) really learn. Make no mistake, though, by assuming that this book is not serious.

1. The Way the World Works is Broken
   1. Planning is Useful, Blindly Following Plans is Stupid
   2. Empirical Basis
2. The Origins of Scrum
   1. Learning to Think Like a Robot
   2. Looking for Something Better than Waterfalls
   3. Inspect and Adapt
   4. Change or Die
   5. Shu Ha Ri

As the name indicates, this chapter is about convincing the reader that a change is necessary — the second one also serves this purpose. And, boy, is it necessary.

Back in the 1990's, when Jeff Sutherland was first transitioning from his academic career, people were overwhelmingly using the Waterfall Method to design software. This method was invented in the 1950's and would prescribe directives to planning software projects: static directives. Managers and directors would carefully lay out almost immutable planning that should be carried out for months and years without knowing what their customer really wanted or what would happen in the meantime. Eventually, the Waterfall Syndrome would be characterized by giant Gantt charts — a sort of sequence diagram from the 1910's — displaying documentation for the sake of documentation, a product that was different from what the customer expected or wanted, and a titanic hole in the company's budget.

The genre of static planning is not dumb intrinsically though. It appeared in an era where, probably due to no feedback to the product developers, engineers would have to use their intuition to guess what the customer wanted. However, it becomes evident that, once feedback exists, static planning is only a very small set of iterative planning, or, in other words, simply inferior. And that inferiority became blindingly clear in one of the biggest flops of the Waterfall series of disasters: updating the FBI Database System.

It was 2010 and the FBI still used physical documents to proceed with investigations. Physical documents with 3 copies — !!!. Any procedure would first have to be approved by some mysterious boss up the hierarchical chain with a mystical red pen in order to get things started. Those were not the 1950's, they were the 2010's.

When 9/11 happened, the biggest criticism of the entities externally auditing the FBI was the sluggish communications inside the institution, making things painfully slow and easily exploitable by terrorists. Following severe Senate scrutiny, the FBI was virtually forced to modernize its system: an initial bill of US$ 170 million. Three years later and nothing had been accomplished. Nothing. People were literally dying because of planning for the sake of planning, beautiful Gantt charts everywhere and no product, no update anywhere to be found.

But don't worry, let's try again. This time we are going to privatize the renewing process, that will surely solve the problem. Let's take a great, big company like Lockheed Martin and everything should be fine. It was 2010 and Lockheed had spent US$ 405 million with only half of the specifications, estimating another 8 years of work and an extra US$ 350 million to get the software completely done. A database update was turning out to become an almost billion dollar, decades lost, gigantic flop: the way the world works is broken.

In preparing for battle I have always found that plans are useless, but planning is indispensable.

Dwight D. Eisenhower

The FBI case illustrates the main recurrent theme of the book and the cause for Scrum's creation: human misinterpretation.

Whenever we plan, we are creating an interpretation of the world that may be either not true or only true for a brief period of time. We must take that into account. The problem is that Waterfall methods or Gantt charts create plans that are not meant to change, they do not incorporate change in their logic. Through iterative methods such as Scrum, we adjust our interpretations with periodic tests, trying to steadily reach the customer's mind.

If you still question Scrum's value, take the extensive research made by (dead serious) companies such as Gartner, Forrester Research and the Standish Group: "companies that still cling to tried but-not-true ideas of command and control and that attempt to impose rigid predictability are simply doomed to fail if their competitors use Scrum". Companies, thus, have two choices: change or die.

In the case of the aforementioned FBI database update, after using Scrum, with 5% of Lockheed's budget, less than 25% of the employees, and less than a 18 months (25% of Lockheed's estimate), the FBI in-house developers delivered a functional product with 90% of the original specifications. If that's not a slap to the face of traditional methods, nothing will be.

One of the other reasons for the downfall of traditional planning is related to the Pareto Rule. It states that, in most endeavors, 20% of the invested time or effort accomplishes 80% of the final value. However, to really nail that 20% gold nugget, you have to really know where the value lies, which is something you will almost never do at the first iterations. If you're spending 5 years to get only 50% of value, like Lockheed Martin did, you have to become aware that you're doing something wrong.

1. People over Processes;
2. Products that actually work over documenting what that product is supposed to do;
3. Collaborating with customers over negotiating with them;
4. Responding to change over following a plan.

Another empirical basis for Scrum's methods is the Toyota Production System (TPS), especially Taiichi Ohno's 8 Wastes. Essentially, TPS is flipping traditional management upside down. Instead of having managers carefully correcting and micromanaging their workers all day everyday, managers will actually be master-servants, whose only function is to get things out of the way of their teams, i.e., eliminate impediments. An example of this occurs at the end of every production cycle or sprint: the manager will always ask the team what can be done to improve how they work, what impediments should be eliminated. By constantly improving how they work, for example, the in-house FBI team initially increased their productivity by a factor of 3 — and that's not really that impressive, great teams can sometimes get up to an 8 times increase. In later chapters, we will delve into a more in-depth discussion about the correlation between TPS and Scrum, dating back to W. Edwards Deming and the revolution the American engineer and statistician caused in the Japanese industry, which later rerevolutionized the world's software industry.

It is not possible to dissociate Scrum's origins from Jeff Sutherland's formation. And, at the core of it is his career as a Korean War Pilot. There, he learned one of the most important methods of his life: the OODA Loop, Observe, Orient, Decide and Act.

The OODA Loop served as an inspiration to the iterative nature of Scrum. It was invented by the legendary Mad Major, John Boyd, as a way of understanding why American pilots were overwhelming Russian bigger, faster, stronger fighter jets.

In the end, after long research, Boyd found that the most important part was guaranteeing constant feedback to the pilot, which would make him able to change his orientations, decisions and actions constantly and accordingly. The feedbacks inside the OODA loops would make quite the mark on Jeff, being essential to the later development of Scrum.

After his career in the military, and having obtained a bachelor's degree in engineering, and earning a master's degree in statistics, Sutherland spent years in a PhD program, and as a researcher in the University of Colorado trying to understand what happens in a cell to turn it cancerous. Years later, when a sudden change in politics would drastically diminish investments at his institution, he realized that he could perhaps apply what he had learned about complex systems to organizations and teams in the corporate world: "how can we figure out simple rules that will guide teams to settle into a more productive, happier, supportive, fun, and ecstatic state?"

In 1983, MidContinent Computer Services contacted him about their hottest new product: the Automatic Teller Machine, later known as the ATM. As was expected at the time, development was carried out using the dreaded Waterfall method, leading to much more than a budget disaster — costs were 30% higher than revenue. Screaming, micromanaging, passive-aggressive behavior and demands for harder work and overtime were common practice. Change or die: "the first thing we need to do is to stop doing stuff that is killing us (...) we've got to figure out a better way of working", Jeff told his superior.

The ATM project would be the practical birthground of Scrum. Ten years later and the best practices developed in that project would be refined into the concepts of the Product Owner, the Product Backlog, Sprints, etc.

A little bit later in his corporate career, Sutherland happened to share the building with some MIT robotics professors and PhD students. MIT didn't have enough space for all of their mechatronics researchers so they outsourced the space. It just so happens that from time to time some of their robots would go into Sutherland's office and chase him non-stop. Fascinated about those machines, he would later ask Professor Rodney Brooks about how they worked. Prof. Brooks's answer would spark some of the key ideas behind the essence of Scrum.

At first, scientists spent billions of dollars and many, many years of hard work trying to build and code the biggest computers and databases. But, at the end, all they got was a computer that could beat humans in chess. There needed to be another approach to building these supermachines. So his team started to experiment with a whole new paradigm. The idea was, instead of building a brain that would control everything, to build modular parts that could think for themselves and, most importantly, interact with the other parts. The key was to make each module's main concern simply not to impede the other ones. And that's how Scrum converged with the Toyota Production System in a way that was both empirical and logical. Another key takeaway from the robot's behavior was it being completely autonomous; there was no master server or hardcoded recipe: it learned to interact with the outside world from scratch and it optimized for its own modules' behaviors.

After his departure from his job near MIT in 1993, Sutherland became the VP of Object Technology — whatever that means — at Easel. The executives at Easel wanted him to develop a completely new product in 6 months for one of their biggest customers: Ford. After a little bit of experience within the company he sat down with his team and told them the bitter truth: they wouldn't be delivering anything if they stuck to Gantt charts and Waterfall bullshit. His later dialogue trying to convince the CEO is quite remarkable:

“How many Gantt charts have you seen in your career?”, I asked.

“Hundreds,” he replied.

“How many of them were right?”

He paused. “None.”

His team spent a few weeks researching about new, better management methods and eventually stumbled upon the most crucial seed of Scrum: The New New Product Development Game, by Hirotaka Takeuchi and Ikujiro Nonaka, published on the Harvard Business Review, in 1986. Directly from Jeff's book — bold characters are my editing:

Takeuchi and Nonaka had looked at teams from some of the world’s most productive and innovative companies: Honda, Fuji-Xerox, 3M, Hewlett-Packard, and others. They argued that the old way of doing product development, typified by NASA’s Phased Program Planning system — a Waterfall system — was fundamentally flawed. Instead, the best companies used an overlapping development process that was faster and more flexible. The teams were cross-functional. The teams had autonomy. They were empowered to make their own decisions. And they had a transcendent purpose. They were reaching for something bigger than themselves. Management didn’t dictate. Instead, executives were servant-leaders and facilitators focused on getting obstacles out of their teams’ way rather than telling them what and how to do product development. The Japanese professors compared the teams’ work to that of a rugby team and said the best teams acted as though they were in a scrum: “(...) the ball gets passed within the team as it moves as a unit up the field.”

At Easel, they had nothing to lose when compared to the mediocrity of Waterfall-type of methods. After six months, and after having spent almost one innovating about how to work, they had managed to successfully deliver the product. Less than two years later, in 1995, Jeff Sutherland and Ken Schwaber would publish the paper SCRUM Development Process, at the Association for Computing Machinery Research Conference, giving formal birth to the method.

As a "minor" anecdote to end this (sub)section, I would like to mention a story told later in the book about a joint venture between General Motors and Toyota.

In 1982, GM closed the New United Motor Manufacturing Inc. (NUMMI) plant in Freemont, California:

Management considered the workforce the worst in America. People drank on the job, didn’t show up for work, and subtly sabotaged the cars (by, for example, putting a Coke bottle inside a door, where it would rattle and annoy customers). Toyota reopened the plant in 1984. GM told them about how awful the workers were but that the managers were great and they should rehire them. Instead, Toyota declined to rehire the managers and rehired most of the original workforce — even sent some of them to Japan to learn the Toyota Production System. Almost immediately the NUMMI plant was producing cars with the same precision and as few defects as cars that were produced in Japan. Same people, different system. GM never reached those levels of quality at any of its other American plants. It pulled out of the joint operating agreement the same year it went bankrupt.

This comes from a discussion farther in the book. Summarizing, the moral of the story is that people are usually not the problem, the system is the biggest source of issues in fact — "look for flawed systems, not flawed workers".

As previously stated, Scrum teams commonly achieve hyperproductivity. This can go from 300-400%, to 800% on very good teams. In order to reach for these impressive numbers you have to guarantee that your team has:

• Autonomy
• Cross-functionality
• Transcendent Purpose

But how do you actually achieve that? Well, I — and Jeff as well — won't be explaining that so briefly here. That's basically the next 200 pages of the book. However, we can at least (very briefly) discuss the seeds of the thinking process.

Scrum comes from techniques used in Japanese manufacturing, but where those come from might surprise you:

Ironically, they learned many from an American: W. Edwards Deming. Deming worked for General Douglas MacArthur during the American occupation of Japan after World War II. MacArthur’s approach to rebuilding the economy was to fire most of the senior management in Japanese companies, promote line managers from the ranks, and bring in business operations experts like Deming from the United States. Deming’s influence on Japanese manufacturing was dramatic. He trained hundreds of engineers in what is called “statistical process control.” The basic idea is to measure exactly what is being done, and how well, and to strive for “continuous improvement.” Don’t just get better once; get better constantly. Always be looking for something to improve. Never, ever settle for where you are. How you get there is to be constantly creating experiments to see if you can achieve improvement.

Deming created, in parallel, a similar reinforcing loop to John Boyd's OODA loops called the PDCA Cycle (Plan, Do, Check, Act) which was and still is taught in management courses around the globe.

Again, I would rather say "Evolve or Die", Scrum is better than rudimentary management techniques, it is not only different. To illustrate that it is really better no matter the circumstances, no matter how good the team is, Sutherland mentions a consulting experience he had at a company called BellSouth:

I learned this firsthand at BellSouth, when I visited them as a consultant years ago. They had top-notch engineers, many from the famous Bell Labs. They executed Waterfall perfectly. They’d bid on huge $ 10- to $ 20-million projects. They’d gather all the requirements from the customer, then go away for eighteen months and deliver on time and on budget exactly what the customer had asked for. They were one of the very, very few companies in the entire world that could pull that off. The problem was, at that point the customer no longer wanted what they’d said they wanted. Circumstances had changed. Business cycles were getting shorter, and customers were demanding more responsive services.

I was brought in to see if I could help BellSouth figure out what they were doing wrong. I soon realized it was the entire way they were working. This can be tough to hear when it seems as if you’re doing everything right. So one day I stood before a roomful of 150 BellSouth engineers and told them that unless they changed to a different, more customer-responsive model, they wouldn’t last as a going concern. The crowd was tough. They were all really smart men and women, but they believed my ideas were just another management fad. I couldn’t get through to them, so I just shrugged and left them with a final warning: “Change or die.” As you may have noticed, BellSouth isn’t around anymore.

What happened is a reflex of the shocking efficiency numbers he mentioned before. If your competition is using Scrum to achieve 300-800% higher efficiency than traditional methods, how beyond mind-boggling spectacular does your team have to be to compensate for that? You just won't find many people like that. And, if you do, you might as well use iterative methods like Scrum.

This is something I've experienced quite a lot in Go (Baduk or Weiqi). It is present in almost all Asian martial arts also, not only Aikido, which is the example Sutherland uses.

Personally, I think that Aikido's teaching methods are incredibly inefficient and the major reason for its unpopular status, not really because it's a bit esoteric and too specific. In this martial art, the master will not teach the technique; at most, he will show you how it's done. The purpose of this is to show the student that learning by doing is the only way to real mastery. The problem here is that, if the master taught the student about the student's specific weaknesses, the pupil might need to practice less on unnecessary content, being able to focus on more interesting or valuable concepts and, perhaps, even innovating on some. In sum, there's a huge waste of expertise in this sport, it feels as if the master were hiding precious knowledge only due to blind tradition and pride.

Anyway, the three Shu Ha Ri states can be found in all crafts, not only martial arts. In my only visit to China, in 2017, in Hangzhou, I received as a gift from some Monks a fan that had a similar inscription on it. I don't quite recall what the ideograms really meant, but it was something like: "first, you see; then you don't; and, finally, you don't need to". Sounds really mysterious, huh? It is supposed to mean that, first (Shu), you think that you know what you're doing, you will follow the master and think that's all there is to it, you're apparently on the direct and correct track. Then (Ha) you start to grasp a little bit of the overarching concepts behind what you were first learning and realize that it was all just the tip of the iceberg: you can't see anymore, it is all overwhelming. Finally (Ri), you incorporate the essence of whatever it is that you're learning, to the point where you are the living manifestation of it, you don't need to see it, you simply are it. From my experience in Go and other crafts, the tougher the art you're trying to learn, the more cyclical Shu Ha Ri becomes, i.e., the more times you will sequentially loop through the three states.

[In the] Shu state you know all the rules and the forms. You repeat them, like the steps in a dance, so your body absorbs them. You don’t deviate at all.

In the Ha state, once you’ve mastered the forms, you can make innovations. Put an extra swing in your step down the dance floor.

In the Ri state you’re able to discard the forms, you’ve truly mastered the practice, and you’re able to be creative in an unhindered way, because the knowledge of the meaning of aikido or the tango is so deeply embedded in you, your every step expresses its essence.

The lesson Sutherland tries to convey with the discussion about Shu Ha Ri is that, although learning the Scrum's "rules" and "recipes" is quick and easy. It will take you a lot of time and, most importantly, practice to reach its Ri, mastery state.

This is supposed to be a series of posts actually. However, that doesn't mean that everything has to be as in-depth and as long as this first post. In the next ones I'll be only focusing on some shorter crucial findings Sutherland exposes in his book. Stay tuned.