A note on reading, writing, and reviewing research papers

Today I came accross Mircea's post on "How to read a research paper". His advice surely is useful all around the year, but in the context of ICSE/Workshop notifications and reviews, the content particularly struck me.

I liked the 3 step structure, as it accounts for the fact that reading research papers is usually not happening in an end-to-end manner (especially giving limited ressources). In addition, it pictures well the iterative process of drilling down to the full content of a contribution. *

In the further readings, a more extensive post on the topic by Bill Grisworld provides a list of questions the reader should pose to a paper.

Two questions came to my mind when looking at this list:

• What would happen, if the authors of a paper minded this list and made sure their products lived up to the questions?

• What impact would it have on our research community, if reviewers 1) judged a paper at the earliest after step 2 of Mircea's reading process** and 2) structured their judgement according to Bill's questions?

My guess would be that, both sides contributing their share, we could get to the point of constructive feedback much quicker, so I'd like to invite everyone to give it a shot!


* Personally, the time boxes assigned to the steps are appealing to me, as I can use them as benchmark for my reading effort.
** I guess that this should be a sort of minimum requirement. Often, I feel, skipping step 3 really isn't doing justice to other's work.

ICSE 2013 panel on Technical Debt: Past, Present, Future

One of my favorite sessions at ICSE 2013 was the panel on Technical Debt: it fostered lively discussion, attacked very relevant questions with respect to practice, and clarified several aspects I had been wondering about.

Since over the conference I overheard several people regretting to have missed the panel (or couldn't get a space), I decided to rework my notes and summarize the most interesting points in the following post. Enjoy!


On the panel: Ipek Ozkaya, Barry Boehm, Alexandra Szynkarski, Judith Bishop, Phillipe Kruchten, Pradeep Kathail 
Moderation: Steven Fraser
In the audience: a balanced mix of participants from academia and industry.

What is Technical Debt?

Many questions touched upon the definition of technical debt, which seems to be a straightforward metaphor, yet elusive (maybe another thing it has in common with real debt?). So how should one determine whether something is "just" bad quality or "real" technical debt?

Phillipe Kruchten: Technical Debt is not referring to externally visible defects, that's bad quality. Technical Debt is the internal bad quality that slows down the development, not in the face of the customers, but the developers. There are also Technical Debt aspects unrelated to quality, e.g. a "technological gap", technological decisions that turned out bad in retrospective. Maintainability and Evolvability sort of capture Technical Debt. Technical Debt is also different from "things not yet done". Technical Debt occurs when, for getting things done, you start taking bad shortcuts. 

Ipek Ozkaya: There is a lot of confusion about Technical Debt. If it is Technical Debt: tell when you will pay it back.

 

Can Technical Debt be avoided in practice? And how should one ideally proceed when faced with Technical Debt? 

Ipek Ozkaya: Sometimes short-time decisions need to be made: track those (which is not happening currently), to make conscious what is happening. Separate Technical Debt from defects and other development tasks that are happening. Improving the visibility of Technical Debt might be really helpful. 

Barry Boehm: Parnas on designing software for extensibility: encapsulate sources of change into modules, to avoid Technical Debt

 

Does Technical Debt materialize itself in code?

It would be practical to have structural measures that correspond with Technical Debt. However, only certain aspects of Technical Debt manifest themselves in the source code. Others are less probable to be detected on the code level.

Phillipe Kruchten: Propellation is one of the indicators proposed: When you do changes in one part, what are the effects on other parts? Another metric, proposed on the workshop is betweenness centrality. SIG and others have their own proprietary metric. There are various static analyses, detecting smells.. it is still work in progress and needs to be validated. 

Ipek Ozkaya: However, the measurement doesn't necessarily indicate the source of Technical Debt. 

Alexandra Szynkarski: We measure changes between snapshots, complexity etc...

 

How should you (economically) quantify the negative value of Technical Debt for decision makers?

Ipek Ozkaya: If it gets to the decision level, Technical Debt is already causing problems that defect the business value. It therefore has economic consequences that can be communicated. It might even be fine to not touch it; think of the impact on business goals.

Alexandra Szynkarski: Match the message to the audience, managers like $$. Map Technical Debt to the quality goals that are hampered by it. 

Phillipe Kruchten: Quantifying Technical Debt is hard, as for other aspects of SE. What is the value of software architecture? Why should we pay expensive people to do that? We know what the cost is, but the value is invisible to the users... another instance of same problem with Technical Debt.

 

How should Technical Debt be addressed?

The current effort-time dynamic seems to restrict the possibility to remove Technical Debt: development moving to regular release cycles leave little opportunity to adjust the effort dimension. Feature content has impact on release quality.

Phillipe Kruchten: In practice, Technical Debt cannot be tackled in isolation due to fixed resources. We need to find balance between bringing new features, fixing externally visible defects, and reducing internal Technical Debt. The question is: How much Technical Debt (and which) do I need to fix to keep new features and bug fixing at a good pace? Technical Debt needs to be kept visible! Developers usually know about it, but it is not propagated up to decision makers. 

Judith Bishop: The software world is intensely competitive, keep this in mind! It is necessary to keep features coming to stay alive... figure out if you can afford fixing Technical Debt. 

Pradeep Kathail: The industry has accumulated a lot of Technical Debt to enable customers and it sometimes is the right thing to do if it is well understood. The biggest problem is that Technical Debt is not documented. It is difficult at higher hierarchies to plan for Technical Debt fixes. Maybe dedicate one release for removal of Technical Debt can work better. 

Ipek Ozkaya: Different measures might be adequate for different Technical Debt items (check Cisco's Technical Debt workshop contribution). 

Phillipe Kruchten: There are different strategies for addressing Technical Debt: It is like changing the carpet of this room by sending everyone outside in a coffee break, or changing it row by row, trying to keep the panel running...  

A lot of Technical Debt is accumulated in historically grown systems. How should one foster the mindset of individual business units without the immediate benefit being visible?

Pradeep Kathail: At Cisco, many different business units were mucking around the code base, slowing down the company. We figured out what was important to the business units and for the company. The conlusion was: Our goal is to satisfy the customers, cleaning up will help to fulfill this goal better! 

Phillipe Kruchten: So far, the focus is on Technical Debt in code, but Technical Debt lies also in the context: the way of building the code, the mindset of people, on the tool level... (connects to social debt?) 

Ipek Ozkaya: You can find one instance of this problem in the workshop proceedings: What happens when one team created Technical Debt and another one needs to pay it back? The same problem occurs in product lines: Who pays for the globally created assets? 

 

How do we develop good strategies to partition different Technical Debt and address them accordingly?

Phillipe Kruchten: There are two big blocks: on the code level, Technical Debt can be assessed and drawn out; its presence is usually known. On the other extreme, Technical Debt can occur in the architecture, the technology choices.. This type of is not evident, you might be unaware of it and the interest is unknown; a fact, which breaks the metaphor of debt. A lot depends on what we want to do in the future... 

Ipek Ozkaya: With explicit Technical Debt, it is rather easy to make a plan. Implicit Technical Debt needs to be traced down to the origin. Track down what is going on, then devise alternatives to go forward. 

Pradeep Kathail: Generally plan for 2,5 times of project time.. a lot of Technical Debt only gets visible when system ships! (Brooks: plan to throw one away, you will anyhow... Mythical Man Month) 

Barry Boehm: Beware of one-size-fits-all solutions! A deep contextual understanding is required to make a good decision. (Example: merger between different companies, which entails unifying the respective IT structures. It could not be decided which solution to switch to on a company-wide scale. Instead, decisions had to be taken on a project or division level.)  

We looked at many organizations that measure Technical Debt for customers. Nasa has a target team that makes sure Technical Debt is tackled.

Pradeep Kathail: Processes are great, but in industry balance is needed between Technical Debt and speed. We need to find out how to transfer processes without slowing down too much. 

Phillipe Kruchten: Some Technical Debt can be completely removed with tools, but other things of Technical Debt (dependent on future) can not be removed with processes. 

Barry Boehm: ESA incentivizes contractors on downtime, social motivation to keep Technical Debt down.

 

What about Technical Capital?

Modern software ecosystem provides enormous technical capital. How does this impact Technical Debt?

Phillipe Kruchten: open source software helps to provide value rapidly. Technical Debt as investment a company does, community has not yet really looked into technical capital.

Thanks, Felienne, for your helpful feedback on the post!

Follow up to "The Connection between Movie Making and Software Development"

I just found this related post by Martin Reddy, a former Pixar engineering lead. Enjoy!

Lessons from Pixar: Why Software Developers should be Story Tellers

Furthermore, I thought I'd share a few animation making trivia facts with you (still from Tony DeRose's talk):

It takes about 4 years to make an animation movie; half of the time goes into story and design, the other half into implementation.

Character design is done by means of a broad variety of techniques, from pen and pencil, over oil to digital. The results are actually shown in an record-breaking art exhibition which is traveling the world. (The same is planned for the science behind the animation.) Personality poses of characters are made in 2D and 3D (e.g. as clay models which are later laser scanned and used to check with the 3D model). Concept art is produced for the environment.

Tony DeRose (TDR): "It still amazes me that we managed to make an appealing story about rats in a kitchen!"

In a process called rigging, virtual controls are added to the character to enable motion and other manipulation (e.g. the degree of movement for eyebrows, and more... there are about 300 controls in a character's face, compared to 30 in a human's face. In total a character might have several thousands of controls.).

Shading, the process of putting textures unto characters and objects, is a lot of work: "Look how much stuff there is! Somebody's got to model and place all these objects!" Especially realistic textures that are not smooth but e.g. feature dents, are very hard to make, but vital for the realistic feel of the film as they give each item a back story. "Next time you see one of our films, step out for a moment and appreciate just how much stuff there is!" (TDR)

Lighting artists position the virtual light sources to ensure that the light hits the virtual cameras exactly right.

"Hair is essentially another character in the film" (TDR)

With a physics simulation layer, it actually matters what happens to a character off-screen.. So if you downsize it too much, it might loose the items it's carrying...

The Connection between Movie Making and Software Development - Tony DeRose, Pixar

There are several reasons why I very much enjoyed Tony DeRose's keynote on the connection between movie making and software development:

It gave great insights into the process of movie making at Pixar (unfortunately the technical ones exceed the scope of this post) as well as getting across one important point: make sure to get your story right, be it for movie making or software development!

Particularly their form of organization struck me as very transferable to software development: Pixar movie development is organized in small teams, with clear roles and responsibility for product vision (director) and project (producer). The entire team works together closely, finding and refining stories, animating the shots, reiterating, etc. to make sure the result is living up to the directors vision.

So how exactly do they organize to achieve this goal? Regular, even daily, meetings are used to pitch stories, storyboard drafts, review animation or designs, to refine, rework, discuss, adjust the work to the vision of the director. The current state of the work (from sketchy to fully animated, very abstract to detailed) is packed into a story reel (here's a progression reel from Toy Story), which plays the movie-to-be, highlighting any (story) flaws, that might get covered up by fancy effects otherwise.
There's a lot more to it until the movie is finally to be released; however, the key point for this successful process to me seems the (let's say it!) Scrum-like clear definition of responsibilities as well as the continuous feedback and review sessions.

Interestingly, when taking on replacing their 80's dated animation system, Pixar first turned to "typical" engineering approaches, which resulted in two failed attempts.

At that time, Steve Jobs proposed to consider the software development process more a creative process then an engineering process. As a consequence, the next development attempt was structured according to the Pixar movie development organization (with slight modifications, such as changing from daily reviews to "engineering weeklys"), using the established tools such as pitching, story reels, and continuous iterative refinement.

Turns out, this way of tackling the problem was really good for the design phases, but did not provide enough support for project planning in the implementation phase (so they took to "something agile", whilst keeping up their reviews and pitches).

Remains to be said that despite the usual pains during roll-out, the system has become very successful, enabling many things that before could not be done (check out Merida's hair in Brave; it's so hard to do that "hair is essentially another character in the film" (seems the problem lies mainly in the"hair-to-hair collisions"!)).

To sum up: movie-making techniques such as story-telling, pitching, and progression reels can greatly improve software development. And, as was remarked during the Q&A, presumably also scientific paper writing!

Why I enjoy attending conferences (some sort of "About this blog")

I guess, everyone right now is expecting the usual perks of conferences: great locations, (hopefully) nice hotels, free food, networking... that's all nice, of course, but not what I am aiming for right now. 

Essentially, conferences inspire me - the people, the interactions, the talks. So each time I'm attending, I'm setting out to try something new. This time, my choice fell on taking up a blog on my area of research, Software Engineering. Possibly also thoughts on teaching might end up here from time to time.

With a lot of new ideas come to my mind and after collecting an awful lot of notes during some of the sessions of this first day of ICSE 2013, I found myself reluctant to bin them into my usual folder (where they might or might not get read again) and instead ready to attempt refactoring them into a blog post (which will hopefully follow this one!).

To sum up, the idea of taking up a professional blog has been lingering around the back of my head for a while. Now, also due to the encouragement of Felienne, it is coming to action.

Let's see how it goes, stay tuned!