#Why We Need #Agile #Agility for #DevOps

<Architect for Developers>

Increasing vehicle speed should not increase rider’s headache.

Who needs quickly produced untested and insecure SW products?

My reader might be surprised – why have I returned to the topic of Agile and DevOps after all these, more than 10, years of these practices? Here is no secret – I’ve read the statement: “Although in principle it is possible to practice DevOps with any architectural style, the microservices architectural style is becoming the standard for building continuously deployed system”. As we know, the reputation of Microservice Architecture is shadowed by distorted Microservices and problems in Microservice protection where convenience and velocity of Microservice development contradicts several Principles of Agile Manifesto. I’ve decided to look into the roots of DevOps to find what is happening with Microservices in there.

The etymology of DevOps is this:

[Previous amendments to Waterfall] – Lean – Agile Method – Agile Manifesto – Scrum Guide – DevOps.

DevOps have inherited automated builds and tests, continuous integration and delivery from Agile Method. They also adopted acceptance of small and frequent changes from Lean Management where continuous delivery was assumed as the fundamental factor delivering value faster, but the focus on value to the end customer had been lost somewhere on the way. DevOps also try to conquest the right to decide whether the SW product is in the releasable state or not. Some tools used by DevOps for automation “trust” the developer that all necessary code, quality, security and compliance controls had been performed and passed, and the outcome can be deployed for the use of end-users. Apparently, this is an ideal picture while in reality, tests, security and compliance controls are omitted because a developer does not know how to execute them and they slow down the delivery.

While DevOps have elevated delivery velocity to the level that only Sun is higher and made it the goal and work effectiveness criteria (regardless quality), they still claim they are agile. I’ve taken a look into Agile Manifesto and Scrum Guide for the role of velocity. Neither the Agile Manifesto nor the latest version of the Scrum Guide (2017) reveals velocity at all. Only Lean mentions velocity but points to providing “continuous delivery with high software quality”. The latter does not fit with test recommendations for Microservices.

For the last 10 years, the software development context has dramatically changed. When Agile methodology was focused on communication gaps between customers and developers. It is apparent that applicability scope of Agile Method was inside the enterprise only – no company but a start-up allowed developers to reach out to customers in the outside market. The DevOps have shifted the focus on the gaps between developers and IT operations / infrastructure, including deployment of developed software. Unfortunately, a goal of automation has obfuscated to DevOps the massive work needed to approach the product development and product release.

As an example, a traditional practice of creation of SW products for the market includes several fundamental steps such as:

1. Identifying a need for technical solutions for a particular business problem. For example, until an Adaptive Case Management for the given business task became a commodity, it had to be executed manually. Actually, there is a hope that AI would help to automate this mechanism before it commoditised
2. At a high level, identifying IT systems, applications and technologies that can/will address the need
3. Identifying what is the company has to build versus reuse versus buy for the solution
4. Adjusting Technology Roadmap
5. Defining development programmes and splitting them in the projects. This means defining goals and objectives of each of them
6. Coming up with an integral solution for each programme and some, more complex, projects.

You can argue that we have a lot of SW products created by one or a couple of people and they can keep all aforementioned steps in their heads. Yes, but it is good (in average) for only very small products. I hope that you do not promote an idea that all SW products will be that small in the close future.[1]

In reality, bigger software products will requires some of listed steps (the bigger product, the more steps). One of the major aims of these steps is to identify and mitigate market and development risks as much as possible and split complex tasks into smaller ones before passing them to the DevOps teams. The production time activities include a few levels of support, maintenance including refinement and retiring. If DevOps Team’s members take responsibility for these works, they will stop producing new SW products and it will happen rather sooner than later.

Thus, dexterity in agile/DevOps development/delivery is not enough for industrial production of software products.

I am inviting my dear reader to join me in the review of agile methodology in conjunction with DevOps and create new values and principles for DevOps from the perspectives of their employer, its business values and industrial marketability[2].

 

Agile Manifesto in new dynamic business and technology context

Contemporary market is characterised by the high and even accelerating flow of changes stream from business and technology as well as from Government attempts to regulate both of them. Certainly, the understanding of things changes over the time and the new understanding can radically differ from the initial one. Let’s take a look at the Agile Manifesto first.

As we know, Agile Manifesto comprises 4 values and 12 principles. When I’ve put them against current practice (still claimed to be agile), I’ve got that all of the values and 8 principles call for rethinking and re-interpretation. Below, there are just a few examples; the full table of comments is available in the Appendix.

• “Working software over comprehensive documentation” [Agile value]

Comments: the time when the Malifesto was written, a product stood as-is in the use for some time, i.e. could be used with minimal documentation. If the developer left the company, the product usually was discarded because could not been supported without documentation. Nowadays, the job market is much more dynamic and company cannot afford product dependencies on the developer. Documentation becomes essential.

• “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.” [Agile principle]

Comments: a term ‘working software’ requires an explicit definition, which has to answer following questions: 1) who decides if the software is working or not? 2) who decides may running code address only ‘sunny day’s scenarios but fail on the ‘rainy day’ scenarios? Is unsecured software a working software? Is incompliant software a working software? Is a barely tested software a working software? Without answering all these questions it is impossible to define ‘frequent’ delivery.

• “Continuous attention to technical excellence and good design enhances agility.” [Agile principle]

Comments: it seems that current practice (after 10 years) substitutes “technical excellence” with the requirement for velocity in delivery in spite of this mainly deprecates quality; “excellence” has become totally offended and disrespectful.

Scrum Guide loopholes

“Scrum is an agile process framework for managing complex knowledge work, with an initial emphasis on software development. Scrum “is designed for teams of ten or fewer members, who break their work into goals that can be completed within time-boxed iterations, called sprints, no longer than one month and most commonly two weeks, then track progress and re-plan”.

From the first step, I have to struggle with ambiguity: who decides whether one “knowledge work” is complex or not. It is definitely easier to apply Scrum to everything…Also, people in the Scrum Team are supposed to “break their work into goals that can be completed within time-boxed iterations”.

Well, are they capable in all cases or they should only estimate the number and duration of time-boxes/Sprints? Who breaks the business tasks for the Teams? Who controls the implementation integrity of preliminary broken business tasks? What the Scrum Team knows about business and technology execution context of the company and market trends related to the given sub-task? All these questions define the scope and complexity of the task that may be given to a Scrum Team.

We know that many practitioners are unhappy with Agile Method and  have attempted to improve it, e.g. by presenting a Disciplined Agile delivery that simplifies Scrum procedures and enables scaling. A lot of development teams are involved in solving relatively complex business programmes (including myself) across the company. They need to scale Agile Method to the a corporate level and  a SAFe methodology has been introduced. In my practice, I’ve found that if number of Scrum Teams in the project reaches or exceeds 3 teams, serious design and implementation (and following support) problems raise with no solutions pointed in the Scrum Guide.

Modern Scrum is interpreted as a framework that includes all different specialists to be administratively placed into the Team.  This is a management fallacy – Scrum Guide talks only about capability of the Team to solve its organisational and operational tasks by themselves; no administration is mentioned. It is anticipated that the Team-members would teach each other and finally ‘cook’ a team where everyone can do ‘everything’ and do all of it professionally, which is an illusion – everyone in the Team is paid for individual contributions and professional unfitness.

It is a very common model where developers are in majority in a Scrum Team. The Scrum Guide insists on collective work on all tasks of the team. However, many tasks are about architectural design, business analysis, quality assurance, operations, security and compliance controls and infrastructural adjustments. Since Scrum Guide does not prohibit ‘democratic voting’, the majority always will be on the developer side regardless if they understand the tasks/problems of other specialities in the team or not. That is, the influence of developers overtakes the influence of any other specialists in all tasks, including work complexity/time estimate, prioritise, task compositions, readiness status and meaning of ‘done’. I saw such cases on many occasions and they resulted in the problems for the team specialists, conflicts and degraded quality pf the outcomes. Software development is based on Computer Science and there is no place for ‘democratic voting’ of inexpert people. I am sure that many people are lackier than me and their experience is much better in the Scrum Teams, but I tried and got much better results when specialists daily contributed in the team-work on their own terms.

The latest version of Scrum Guide (2017) is still full or ‘open ends’ and useless ‘big words’, which unseal ‘deep rabbit wholes’ – options and opportunities for performing ineffective and even harmful activities. One of the biggest problems for Scrum Team is an absence of estimation methodology or process recommended to the teams. The immediate result of this whole is uncertainly in the content, size and details of the Product Log.

Scrum practitioners are proud of their approach to solving a complex task by performing continuous Sprints set for small elements of the task. On this ground, they deny a necessity in having the whole high-level end-to-end view on the complex task. This is either an uneducated or marketing blunder: it is more likely that a small element of the task is wrong (disconnected, out of context and dependencies) if the entire picture is unknown.

Since only Product Owner is exposed to the outside-team world, capabilities of this person to properly see the whole picture and to extract a small element from it are not guaranteed and not systematically supported in the Scrum Guide. Thus, the Team’s decisions on the complexity (typically in User Story points), size, order, priority and dependencies of Sprints are unreliable, especially when the majority of the Team are not specialists in any of enumerated aspects.

The Scrum Guide does not even mention User Stories and their formula – ‘As a <role>, I want…, so that…”. The specific design of the Product Backlog is empiric. If the Team gets an unknown previously task, its entire planning based on the Scrum Guide is erroneous. This is why an IT has to keep special management forces to control and correct the Scrum Team planning, which does not seem like Agile.

The Scrum Guide does not distinct between responsibility and accountability, i.e. it is weak from the manageability side. Many believe that the whole team is responsible for the outcome while no one Scrum Team member is accountable for it. The Scrum Guide leaves this fundamental question open though states that the “Product Owner is responsible for maximizing the value of the product”. Well, note that loophole – “maximizing the value of the product” is not the same as delivery of the product. Also, “Individual Development Team members may have specialized skills and areas of focus, but accountability belongs to the Development Team as a whole.” Thus, since the whole Team is accountable to the consumers and responsible for everything the team has done, no one member of the Team is responsible or accountable for personal work. Nice! This is why team-work like this is not counted in the individual performance assessments, i.e. in spite of all ‘big words’ of Product Owner and Scrum Master, members of the Scrum Team are not motivated and the Scrum Guide is fine with it.

Here are a few more examples of Scrum Guide inconsistency.

1. Product Backlog is under the supervision of a Product Owner. While many suggest that User Stories are the type of content of the Product Backlog it is not completely true. Scrum Guide does not uses a term “User Story” and operates with the term “requirements”. The latter may be functional User Stories – and non-functional – performance, throughput, security, compliance, etc.  As these two types are not explained, many Scrum Teams miss non-functional requirements when working on planning or misplace them into inappropriate Sprints
2. A requirement is a text (formatted or not), which presents what the product team requires or wants, according to inherited Agile practice. Scrum Guide does not say anything about this as well as it does not explain why a Scrum Team wants anything? What the assumptions for the team wanting anything? Where are goals and objectives of the product/project that represent the filter of what may be wanted?
3. Scrum Masters “Help to … protect the team from external interferences” i.e. isolate and disconnect it with reality, as well as “promote a good cooperation between the team and stakeholders” who definitely influence the team. Ends do not meet in Scrum Guide. Also, Scrum Masters “facilitate common sense within the team” while professional sense is not common in many situations, i.e. the value of mixture of different specialities in the Team can be easily lost (which I witnessed).
4. A Product Owner “represents the business or user community” while the Team decides on the delivery and delivery integrity plan. Because of Product Owner’s representation, the Team members are not supposed to interact with business and user community – this is how the Scrum Guide can be interpreted.
5. The Scrum Guide says nothing about how the Scrum Team is created in the first time. Who takes care for checking professionalism of Team-members; who reviews the psychological compatibility among members?

All these questions call for answers if we want a DevOps team to be effective.

I can assume that DevOps have some of mentioned principles with related problems and created new ones. For example, I am for automations of commoditised operations with both hands. Automation is the major mean for reducing human errors and time needed for producing a new product. However, not everything can be automated at once and at current maturity of tool market. If testing more complex than Unit-testing or connectivity/interface testing is needed, e.g. regression or end-to-end, and such testing is simply omitted or security controls and regulated information content and data protection are ignored, I’ll use the same both hands against such “speedy development”. It is not a type of products business wants, it a fallacy, not professional development. If DevOps cannot produce quality products from the business and consumer viewpoints, these products are not for DevOps, simple as that.

Architects and Managers hear almost the same response when questioning DevOps about the product readiness for the market or internal consumption. This answer is: “We need a quick feedback from consumers and we will change the product appropriately. Since we can do this really quickly, we use this mechanism to produce products of needed quality”. I would recommend buying into this to nobody – technology does not know how real consumer-producer relationship works. After the first version fails in the consumer hands – regardless if it is an internal or external consumer – the reputation of the producer drops significantly or even totally. There is no second chance for producer or it will cost him a fortune (to restore the reputation).

A complex product may have 3 features out of promised 7, but each delivered feature must work as specified. Some consumers might accept this shortage due to high integrity and quality of those 3 features. However, if 5, 6 or all 7 features are delivered and none of them is properly working, not secured or not compliant, it is more likely than not that such producer loses the consumer. Described behaviour is known in the market as ‘spitting in eternity against the wind’.

---

[1] At one moment of Chinese Cultural Revolution the Steel Industry was destroyed and replaced by steel melting in house blast-furnaces.

[2] If a DevOps team quickly produces something that only they see a values of, such practice may not be appreciated unless the company does not count its money [researches and POC are obviously out of this matter].

‘Leave a comment’ BELOW WORKS ONLY IN MOBILE VERSION

WorlPress HAS A BUG IN THIS FUNCTION