Software and Peer-Review : The Rousseau Case.

Brendan and Ronald Rousseau’s article with the downloadable software is an exciting and long over due experiment for Cybermetrics. However, it raises an interesting question. Should the software and user documentation be subject to critical peer-review ?

In general, software has two primary components : (a) the program (algorithms and user interface) and (b) the documentation. Software is usually a standalone, self-contained tool with reference material and help files. Since Cybermetrics is a peer-reviewed e-journal its readers likely expect that all of the parts of a journal article including the appended software are peer-reviewed. A critical review assures the reader/user that the software is robust (i.e. based on sound theory and practice), accurate (i.e. the output that is produced meets the author’s claims) and user-friendly (i.e. the user interface is clear and intuitively understandable). A critical review of the software documentation assures the reader that all of the information including theory, methods, limitations and user interface (i.e. such things as menus and buttons) are thoroughly discussed and clearly explained.

It seems that the reviewers of the Rousseau article might have overlooked some of these important factors. First, the only documentation that is available to the user is a brief and seemingly hastily written few sentences accessible through the info button on the user interface. Perhaps the reviewers thought that the web article would be sufficient documentation since it could be downloaded in pdf format. However, the article itself seems to lack a thorough discussion of the theory and techniques implemented in the software. Although there are references to earlier publications in the article it is usually appropriate to include the details in the software documentation and not just the references. The user needs to be able to clearly understand the usefulness and the limitations of a program. After all, the program is a tool not just an academic article.

Second, the authors claimed that « This power law distribution has two parameters : ß and C. These parameters are, however, not independent but are related through the requirement that …. equation (2) ». In general, there is no relationship between exponent, ß, and intercept, C, for a power law having the general form y = C x ß . In fact, for any constant value of ß the value of intercept is free to range from +¥ (infinity symbol) to -¥ (infinity symbol) with the exception that C=0. In fact in some instances the intercept has a significance that is quite independent of the exponent. In this particular instance, it appears that the Rousseau algorithms are restricted to the special case where ß and C are not independent. The implications of this important and sever limitation are not discussed in either in the article or in the program

I suspect most readers of Cybermetrics will urge it to continue with its bold venture into providing useful software and perhaps even articles with appended digital data sources. However, I expect they would also encourage the reviewers to turn a slightly more critical eye to the details of the article, the software and the documentation.

References

Rousseau , B. and Rousseau, R. (2000). LOTKA : A program to fit a power law distribution to observed frequency data. Cybermetrics, 4 (1), paper 4