Disclaimer: Dieser Thread wurde aus dem alten Forum importiert. Daher werden eventuell nicht alle Formatierungen richtig angezeigt. Der ursprüngliche Thread beginnt im zweiten Post dieses Threads.
On “General feedback on summary homework”
Responding to Prof. Riehle’s mail on “General feedback on summary homework” there are two things I want to add:
-
Thank you for pointing this out! If you remember homework 8 (“Twelve Ways to Fool the Masses When Giving Performance Results on Parallel Computers”) I tried to categorize the “cheatings”, but half of the reviewers complained that I haven’t listed all 12 issues. (Of course, my categorizing itself was highly discussable!)
-
Beside the Santini Paper was very entertaining, it was not obvious to me, that it was fictional. This was the first issue to me. The second problem I had: I haven’t read all these papers. So if the reviewer complains about the missing implementation of Codd’ theory or the missing motivation of the work of Shannon, he is maybe right - therefore I would have to review the review and read the original article. So if we are looking for high level abstraction, my conclusion could be like “Even outstanding papers can be rejected and even the best scientists are not perfect.” Otherwise, I think, I should have to write much more beyond the scope of a one page. This is why I think this “paper” was not easy.
In addition to that, I would like to do more in this direction: Summarizing papers is important without a doubt (and quite hard I recognized), but it would be also interesting (at least for me) to classify/judge a paper (not only a colleague’s summary): Is it a good/trustworthy paper? Where are the weak points? What is unclear? Does it follow the standards of research papers?
1 Like
I have a response to the feedback and something to think about, too:
In my first reviews I tried to be more abstract and to omit things in my summary, that I personally found not as important as other things in the original paper. This lead to students, giving me bad ratings which is leading to frustration or at least… “getting bored”.
Further, I think the decreasing quality of summaries goes along with the massive amount of papers to summarize for NYT. It becomes a “routine” to crossread papers (which is a valuable skill of course) and just getting this a half to one page per paper to submit to crowdgrader. Personally, I would reduce the paper summaries and do a little more discussion in class. Maybe one time a homework could be “write a good abstract for the following research question”, and another time it could be one summary again. Just bring more variety in the homework.
I have chosen NYT for the project, and I enjoy thinking about and writing the paper, learning how to latex and how to do proper research for my master thesis. But the summary part is, honestly, really no fun for me. It is boring, all the same, and frustrating with Crowdgrader. I got reviews that rated me “2” for using a bad font without any other critical or helpful content in the review. (I mean - wtf?) Nailing a thesis is more than being able to summarize papers, isn’t it?
If this was an evaluation, my opinion is split: I really like the project part of NYT, but I would suggest to change something on the summary system and drop crowdgrader. If the amount of summaries were lower, there might be no need for a crowdgrading system. Just my honest opinion… I’m going on with summarizing and reviewing now 
1 Like
We are looking at the feedback that students give via Crowdgrader as well. If you get some useless feedback about the font, the person who writes that feedback is doing more of a disservice to him or herself than to you (unless, of course, your font really is so bad that it makes content unreadable).
Being able to summarize material effectively is a skill that you need for academic work, but Michael correctly identifies that being able to evaluate the quality of a paper is important, too. This is necessary both when you are doing your own research and want to decide how authoritative a paper you’re citing is, as well as if you have a career as an academic, where you will be asked to perform peer review. You are already trying this skill a bit when you perform the Crowdgrader reviews.
In my unimportant opinion this is not completely true, because there are three main differences between reviewing summaries and evaluating papers:
- For reviews you know the „grounded“ data sources, this makes it easier.
- Reviewing a paper goes way beyond of checking if there is everything what is „important“ (whatever this means): Has it the right structure (which is more normalized in papers, I think)? Are the results valid? Under which assumptions? Does it fulfill the requirements we now know (precise language, sufficient related work, etc.).
- But I know this is not easy to do, because you will need way more effort and time to review a paper, because you had to go in much deeper. On the other hand, if we would present our review results at lecture, we could could also get a deeper insight what others would criticize and how they review. Additionally, useless review become more obvious.
And it would make it more interactive as it is (I love to argue and discuss. 
)
My suggestion is to have a homework (instead of x summaries), where we write a paper review. It would be nice to have a bad paper (fortunately there are so many out there ) and over two to four weeks we can review this paper. A bad paper because it is much more motivating to find something. This could also add a new perspective to NYT (10ECTS): We actively write a paper and read a review of that, but also have to read a paper and write a review. It would be nice to present the results in class (I think we will do this for our papers next year, but also for the reviews). This would make the lecture much more interactive, which I like it very much. Personally, the discussions in class for each homework are much more helpful for me and my understanding of a text, than the CG reviews. So this should stay!
2 Likes
Hello everyone,
thanks for your concerns. I see a number of different related points that I want to address.
- Quality of student summaries.
I addressed this in my previous email. If you are only crossreading the paper, you are doing something wrong. You will not get the expected learnings out of the paper. One reason why I’m doing the summaries myself is to guess the amount of time it takes. I finish in about half of the time allocated for students for the class (5 ECTS) so I think you should be able to do it in what is considered the required amount of time for the class.
- Amount of summarizing work.
We limited it to two papers and it is getting easier, so I thought it would be OK. In addition, this is a drill you really need if you want to go into research. You’ll be doing it all the time.
- Quality of student reviews.
The quality of student reviews strongly depends on how well they understood the paper in the first place. For that, they actually need to have read the paper properly, extracted properly, summarized properly. If they didn’t, a student’s own summary will not be very good, and any review will not be very good as well.
- Process for quality student reviews.
A good review, as just remarked, starts with a good reading. Then extraction, summary. Revising from the class discussion. I could possibly provide my own structured notes for those who are lost, but I thought students should be able to do it by themselves.
- Relevance of student reviews for grade of reviewed student.
Poor shallow reviews are very obvious to me. I realize only now, I guess, how upsetting poor reviews by other students can be to you. Since they are anonymous (both ways), poor reviews are not directed at the person behind a reviewed summary, but really at the review. Poor reviews only reinforce that the student did not read the paper well or did not understand it well. But this doesn’t mean I know how to handle a disappointing review, I admit. Let me simply say that it will not affect your grade.
- Papers to be summarized.
We are not summarizing nor reviewing research papers in themselves but mostly teaching materials that I though would be helpful. So I agree that reviewing a real research paper would be beneficial. We had thought initially for you to review each other’s work for this but you balked and we removed. I will think about this over Christmas.
- Summary style, in particular summary vs. critical review.
Related to 6. is that it is correct that you are only summarizing, not critically reviewing. This is a question of what’s really useful to you when. You will not be a reviewer of academic research soon but I guess it would be helpful for some to talk about this and teach this how I do with my Ph.D. students. So let me think about this as well.
I hope this clarifies some of the points.
All the best for the holidays!
Dirk