Recognition Metrics

The Pool profiled in Chronicle of Higher Education

2008-06-10T11:54:00.001-07:00

Andrea Foster writes in the 30 May 2008 issue:

Re:Poste, a Web application that encourages academics to pick apart online articles from the mass media, is only in its infancy. But the program has already generated buzz on a social-networking Web site called the Pool....Re:Poste is one of 600 creative works — games, art, and more — by new-media students and faculty members, most of them on the Orono campus, described in the Pool, which also contains about 2,000 reviews of those works. Starting in June, the Pool will have a much wider reach, as people in general will be invited to add material to the site, rate others' projects, build on their ideas, and find collaborators for their own projects.
The Pool, as yet little known, could provide a new avenue for new-media scholars to do their jobs. Eventually it could play a role in their tenure and promotion as well.
The numbers and influence of such scholars in academe are growing, and they are looking for new ways for their institutions to evaluate them. Books and journal articles alone are a flawed measure of their productivity, new-media professors say, because many of their accomplishments exist only as Web sites, interactive games, or multimedia presentations. The Pool, they suggest, can be one measure for judging their work.

Most of the commentary in the article supports the idea that alternative recognition metrics can be useful for scholars of the Internet age. Gerard McKiernan of Iowa State says the Pool could be a good barometer of a scholar's influence:

"Five hundred heads is better than two in assessing the value of a work," says Mr. McKiernan, who runs the blog Scholarship 2.0, on alternative Web-based methods for scholarly publishing.

Richard Chait of the Harvard Graduate School of Education remains skeptical ("I don't know how you authenticate the value of Web-site hits or what people say on Web sites"). He seems to have missed the fact that all three innovations described in the article--The Pool, Re:Poste, and ThoughtMesh--are tools designed to inject trust into the wild and woolly world of online publishing.

jon

Re: Exclusivity and Heresy | Alternative academic criteria

2008-05-04T13:56:00.000-07:00

My take on parallels between control over curatorial and academic contexts--

Danny Butt wrote [New Media Curating list]:
>
>I can't help but think of homologies back to the idea of net.art as an
>attack on the gallery system. What I think has become clearer is that
>the role of curatorial practice, or the museum, or the publisher, is
>not merely that of gatekeeper as it is often conceived in the net.art
>imagination. It is also about the provision of context that is a
>critical aspect of the entire ecosystems of disciplines and practices.
I agree that curators provide context as well as gatekeeping, but if my fifteen years as a curator in a major museum is any indication, the ability to control the context is even more powerful than the ability to control who gets in the door.

Sure, there are some artists and curators mounting risky shows in alternative spaces. But as long as these efforts are evaluated according to the art market's prevailing hierarchy of value, they don't have much effect on the top of the pyramid.

This was precisely the value of Internet art--not just to produce and distribute art outside the museum, but to establish a different context that wasn't under the thumb of blue-chip gallerists and auctioneers.

Similarly, as Sean Cubitt mentioned, university research is increasingly evaluated according to a monolithic hierarchy that reduces each researcher to a numerical standing calculated from the number of refereed articles times the "rank" of each
journal. This rankism is a pitifully shallow view of the ecosystem required for critical or creative thought, and is one of the "impediments to new ideas and expression" that Roger Malina described crippling the contemporary university.

So how can academics nourish the ecosystem for new media research?

1. Publish early and often. The scientists are doing it (see Mitchell Waldrop's article in this month's Scientific American at

http://www.sciam.com/article.cfm?id=science-2-point-0&print=true). Some folks on this list have already published on ThoughtMesh (http://thoughtmesh.net), which will soon launch a "submesh" feature that emulates journal selections.

2. Negotiate each publication with your press. If the contract your press sends you doesn't explicitly allow you to self-archive your work, write it on the contract and fax or email it back. You'll be surprised at how flexible publishers can be.

3. Lobby your university to upgrade its promotion and tenure criteria for the 21st century. As mentioned elsewhere on this list, Leonardo has been quick to see the need to expand publication opportunities for scholars in the networked age; Leonardo
magazine will soon be publishing the guidelines for new media academics produced by Still Water at the University of Maine:

"New Criteria for New Media" (white paper)
http://newmedia.umaine.edu/interarchive/new_criteria_for_new_media.html

"Promotion and Tenure Guidelines" (sample redefined criteria)
http://newmedia.umaine.edu/interarchive/promotion_tenure_redefinitions.html

I've already received a half-dozen emails from folks hoping the publication of criteria like these will force their institutions to recognize the new forms of research birthed by digital media. If you have your own guidelines or want to contribute
to the conversation, please join the Leonardo Education Forum discussion at http://artsci.ucla.edu/LEF/node/104.

Cheers,

jon

"internet too fast for academia"?

2008-03-07T11:34:00.000-08:00

http://blogs.telegraph.co.uk/technology/shanerichmond/feb08/thurman-ugc-study-is-flawed.htm

Does a two-and-a-half year publishing turnaround render studies of the Web dead on arrival? That's the claim argued in the following exchange over a study of the efficacy of reader comments in online journalism. Maybe Leonardo Transactions can help?

jon

* Posted by Shane Richmond on 28 Feb 2008 at 11:28
....Neil [Thurman] is an eminent academic and experienced in this field and I'm not suggesting that this study is without merit. However, many of the problems he highlights are not problems any more. Some of the problems we have now didn't exist
back then.

Does the internet move too fast for academia?

* seamusmccauley 28 Feb 2008 12:45
Shane - I emailed Neil about this, as I had the same concerns (our own cited interviewee likewise left the company more than two years ago). He was kind enough to share with me a far more up-to-date report. I've read it and it addresses some of the
issues you raise. Alas, I understand that academic publishing cycles mean the new report won't be out until September, when things will of course have moved on again.

Not Neil's fault, I hasten to add - he's doing some great work in this space, indeed some of the only really rigorous academic studies into the subject at all. But the academic publishing schedule he seems to be lumbered with does create these
considerable problems of relevance and timing. By the time his papers come out they are essentially recent histories of the web rather than investigations of the current state of the art.

* neilthurman 28 Feb 2008 14:26
As Seamus recognises, the "problem" you perceive regarding the length of time that has passed since the data was collected, is not of my making, but a result of the fact academics are leant on to publish in peer-reviewed journals (who demand
exclusivity) in order that they and their departments are rewarded--for example with income from the Research Assessment Exercise. Even though the journal that published this paper has recently increased its pagination and frequency, more than 17
months elapsed between acceptance and publication (and more than a year between submission and acceptance).

Some academic publishers are trying to speed up the publication cycle (via initiatives like Taylor and Francis' iFirst), although the promised improvements are "several weeks", rather than the months or even years required.

Web 2.0 means Recognition Metrics 2.0

2007-07-10T04:24:00.001-07:00

This interesting policy change from one of the major Internet ratings companies reflects the way remote scripting has changed the way users interact with Web pages--jon.

Computerworld - New Web metric likely to hurt Google, help YouTube

In a nod to the success of emerging Web 2.0 technologies like AJAX and streaming media, one of the country's largest Internet benchmarking companies will no longer use page views as its primary metric for comparing sites.

Nielsen/NetRatings will announce Tuesday that it will immediately begin using total time spent by users of a site as its primary measurement metric....the change was prompted by a continuing increase in the use of AJAX, or Asynchronous JavaScript and XML, which allows a Web site to refresh content without reloading an entire page, and to the growing use of audio and video streaming.

"It is not that page views are irrelevant now, but they are a less accurate gauge of total site traffic and engagement," Ross said. "Total minutes is the most accurate gauge to compare between two sites. If [Web] 1.0 is full page refreshes for content, Web 2.0 is, 'How do I minimize page views and deliver content more seamlessly?'"

For example, he said, MySpace may have 10 to 11 times more page views than YouTube, but myspace.com users spend only three times more minutes on the site, Ross added. Therefore, measuring total time spent on a site will make it easier for advertisers to mold their ads to how users are actually accessing content, he said.

"On YouTube there will be more ads flowing in based on duration (on videos)," he said. "The more time I spend on YouTube ... [advertisers] will figure out a way to monetize that."

PowerPoint bad for brains

2007-04-09T09:42:00.001-07:00

I couldn't resist sharing this link from Alain Depocas--jon.

Research at the University of NSW, Sydney, Australia, claims the human brain processes & retains more information if it is digested in either its verbal or written form, but not both at the same time. More of the passages would be understood &
retained if heard or read separately. "The use of the PowerPoint presentation has been a disaster," Professor Sweller said. "It should be ditched."

"It is effective to speak to a diagram, because it presents information in a different form. But it is not effective to speak the same words that are written, because it is putting too much load on the mind & decreases your ability to understand
what is being presented."

This new insight clearly puts the recent report about using Powerpoint in Parliament speeches in a new perspective.

[ http://infosthetics.com/archives/2007/04/powerpoint_bad_for_brains.html ]http://infosthetics.com/archives/2007/04/powerpoint_bad_for_brains.html

The h-index and its discontents

2007-03-23T06:32:00.000-07:00

In a post to the OACI working group list, Peter Suber noted that the science index Scopus has begun to use Jorge Hirsch's "h-index" to supplement its other measurements of author impact. The h-index is a number calculated from the number of articles
an author has published at high citation levels.

http://www.econtentmag.com/Articles/ArticleReader.aspx?ArticleID=35680&CategoryID=17
http://en.wikipedia.org/wiki/Hirsch_number

In a simple demonstration of the weakness of such all-in-one numerical metrics, Eberhard Hilf of the Institute for Science Networking Oldenburg GmbH pointed out that physicist PW Higgs has a miniscule Hirsch Index of only 9, despite his having
predicted the famous Higgs boson. The existence of this massive particle would resolve some of the deepest uncertainties associated with elementary particle physics, which is why the physics community has spent millions of dollars building and
designing particle accelerators to find it.

Yet another case for more nuanced recognition metrics than a list of names with numbers next to them.

jon

Infoworld interview with Peter Suber

2007-02-18T16:30:00.000-08:00

Trebor Scholz alerted me to this audio interview with Open Access maven Peter Suber, which touches on both recognition metrics and the Interarchive agenda:

http://weblog.infoworld.com/udell/2006/08/18.html

I was especially interested to learn that Connotea (Nature's science-tagging system) can work hand-in-glove with ePrint (the most popular open access repository software). These sorts of metadata compatibilities make the holy grail of searching
across "dark archives" seem closer to our grasp.

jon

MLA flunks humanities in recognizing online scholarship

2007-01-19T07:25:00.000-08:00

Recommendation number 4 of 20 in the Modern Language Association's December 2006 report on promotion and tenure proposes that:

"Departments and institutions should recognize the legitimacy of scholarship produced in new media, whether by individuals or in collaboration, and create procedures for evaluating these forms of scholarship."
http://www.mla.org/tenure_promotion

While it may sound warm and cuddly to us new media jocks, this recommendation comes as a response to a disheartening study on the absence of such recognition in American humanities departments today:

"The task force was dismayed by a widespread lack of experience in evaluating digital scholarship. More than 40 percent of departments at Ph.D.-granting institutions said, in response to the survey, that they did not know how to gauge the merit of
refereed electronic articles, while 65.7 percent reported that they had no experience judging monographs in that format."
http://chronicle.com/free/2006/12/2006120701n.htm

How can 40% of America's smartest brains in the humanities "not know how to gauge" refereed journals in electronic form? If the process of refereeing is comparable to print, I can only assume the answer is fear of the unfamiliar: the lack of
brand-name recognition for upstart journals, and a lack of experience with new scholarly tools--like, uh, a Web browser.

Interestingly, the MLA prodded reviewers to get used to such tools in promotion & tenure guidelines released as early as 2000:

"Review Work in the Medium in Which It Was Produced. Since scholarly work is sometimes designed for presentation in a specific medium, evaluative bodies should review faculty members' work in the medium in which it was produced. For example,
Web-based projects should be viewed online, not in printed form."
http://www.mla.org/resources/documents/rep_it/guidelines_evaluation_digital

I guess we shouldn't be surprised that the study documented even lower acceptance rates for online scholarship that doesn't look like refereed journals. Nevertheless, the 2006 MLA report counters that:

"in evaluating scholarship for tenure and promotion, committees and administrators must take responsibility for becoming fully aware both of the mechanisms of oversight and assessment that already govern the production of a great deal of digital
scholarship and of the well-established role of new media in humanities research. It is of course convenient when electronic scholarly editing and writing are clearly analogous to their print counterparts. But when new media make new forms of
scholarship possible, those forms can be assessed with the same rigor used to judge scholarly quality in print media. We must have the flexibility to ensure that as new sources and instruments for knowing develop, the meaning of scholarship can
expand and remain relevant to our changing times."

Sounds like it's time for humanities departments to offer a crash course in Slashdot karma ;)

jon

______________________________
What do pigs' wings, alien planets, and computer viruses have in common?
They're all At the Edge of Art.
http://at-the-edge-of-art.com

Nature experiments with Open Peer-Review

2006-06-14T11:51:00.000-07:00

Though Nature takes pains to distance itself from critics of the current peer review system, this is an interesting experiment from a publisher sometimes chided for aggrandizing scholarly publishing (eg,

http://www.earlham.edu/~peters/fos/2006_06_11_fosblogarchive.html#115012224869388294).

Nature's "Web Debate" looks like it has some interesting links, though I haven't followed up on them yet:

http://www.nature.com/nature/peerreview/debate/

jon

Date: Fri, 09 Jun 2006 11:08:43 -0400 (EDT)
From: "Nature Publishing Group"<Nature.Publishing.Group@info.nature.com>

Dear Colleague,

There has recently been much discussion in the community about peer-review systems and possible alternative models. As part of our ongoing commitment to readers, authors, reviewers and the scientific community as a whole, Nature is experimenting
with one such alternative model: [ http://info.nature.com/cgi-bin24/DM/y/hYZ80BfPkN0DHy03ef0EE ]open peer-review.

The purpose of this trial is to test the quality and quantity of unsolicited comments made during an open peer-review process, as compared to the traditional peer-review process with no unsolicited comments. Nature will offer to upload a manuscript
to a preprint server at the same time it is sent out for normal, confidential peer-review. Scientists can then comment on the posted manuscript. When the editor has received the confidential peer-reviewers reports, the open commenting period will
be closed.

This trial will run from June through August. During this time, Nature will offer all authors of manuscripts selected for peer-review the option of simultaneously posting the manuscript on the pre-print server. Participation in this trial is
entirely optional, and will have no effect on a manuscripts likelihood of being published.

The traditional peer-review process has withstood the test of time, and we here at Nature do not believe there is any fundamental problem with the process. However, as new publishing technologies offer opportunities for change, we are committed to
exploring alternative models to better ascertain their effectiveness and popularity. To this end, we will be hosting a web debate on the subject of peer-review during this three-month trial period.

More information on Natures open peer-review trial and the accompanying web debate is available [ http://info.nature.com/cgi-bin24/DM/y/hYZ80BfPkN0DHy03ef0EE ]here. You can also sign up for a separate e-alert that will notify you when new
manuscripts are posted to the preprint server and are available for comment. Simply send a [ mailto: Peer.Review.Trial@info.nature.com ]blank email to start receiving these weekly email updates.

I encourage you to take a moment to read more about this trial, and to actively participate in the feedback process, whether by posting comments on individual manuscripts or joining the peer-review debate. We look forward to your input.

Sincerely,

Philip Campbell, PhD
Editor-in-Chief
Nature

Open access to flu data is a matter of life and death

2006-03-17T10:51:00.000-08:00

I cherry-picked this excerpt from Peter Suber's excellent Open Access News blog--jon.

Glyn Moody, Will Data Hoarding Cost 150 Million Lives? Open..., March 14, 2006.
http://opendotdotdot.blogspot.com/2006/03/will-data-hoarding-cost-150-million.html

The only thing separating mankind from a pandemic that could kill 150 million people are a few changes in the RNA of the H5N1 avian 'flu virus. Those changes would make it easier for the virus to infect and pass between humans, rather than
birds....The good news is that with modern sequencing technologies it is possible to track those changes as they happen, and to use this information to start preparing vaccines that are most likely to be effective against any eventual pandemic
virus....The bad news is that most of those vital sequences are being kept hidden away by the various national laboratories that produce them. As a result, thousands of scientists outside those organisations do not have the full picture of how the
H5N1 virus is evolving, medical communities cannot plan properly for a pandemic, and drug companies are hamstrung in their efforts to develop effective vaccines. The apparent reason for the hoarding - because some scientists want to be able to
publish their results in slow-moving printed journals first so as to be sure that they are accorded full credit by their peers - beggars belief against a background of growing pandemic peril. Open access to data never looked more imperative.

Re: weighted impact factors

2006-02-17T11:58:00.000-08:00

Peter,

Thanks for keeping us up on this recent research.

The distinction Bollen et al. make between popularity and prestige is certainly evocative. I hope I am reading the abstract correctly in that a journal's prestige would be an emergent property of the citation statistics rather than a port from the
journal's offline reputation. If so, then I'd be very interested in seeing how they weight PageRank to accomplish this, as the Interarchive group has been exploring a similar tack (though without the "journal" layer).

If, on the other hand, the authors are merely weighting according to pre-Web factors instead of bootstrapping an emergent measure of prestige, that would be a lost opportunity. We've been there before, as when proponents of the "dot-museum" Web
suffix fell back on traditional definitions of a museum to justify imposing a class structure on the inherently democratic landscape of online art:

http://www.mediachannel.org/arts/perspectives/dotmuseum/index.shtml

Thankfully, the "dot-museum" movement seems to have completely flopped. I can't think of any "prestigious" museums who bought into it :)

jon

Peter Suber <peters@earlham.edu> on Wednesday, February 8, 2006 at 2:42 PM -0500 wrote:
>This is an interesting paper.
>
>Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel, "Journal
>Status," a preprint. Self-archived in arXiv, January 9, 2006.
>
>Abstract: The status of an actor in a social context is commonly defined
>in terms of two factors: the total number of endorsements the actor
>receives from other actors and the prestige of the endorsing actors. These
>two factors indicate the distinction between popularity and expert
>appreciation of the actor, respectively. We refer to the former as
>popularity and to the latter as prestige. These notions of popularity and
>prestige also apply to the domain of scholarly assessment. The ISI Impact
>Factor (ISI IF) is defined as the mean number of citations a journal
>receives over a 2 year period. By merely counting the amount of citations
>and disregarding the prestige of the citing journals, the ISI IF is a
>metric of popularity, not of prestige. We demonstrate how a weighted
>version of the popular PageRank algorithm can be used to obtain a metric
>that reflects prestige. We contrast the rankings of journals according to
>their ISI IF and their weighted PageRank, and we provide an analysis that
>reveals both significant overlaps and differences. Furthermore, we
>introduce the Y-factor which is a simple combination of both the ISI IF and
>the weighted PageRank, and find that the resulting journal rankings
>correspond well to a general understanding of journal status.

Pickering and Noruzi articles on citation metrics (via Peter Suber)

2006-02-17T11:36:00.000-08:00

Peter Suber forwarded these abstracts of recent articles on the limits (real or imagined) of Web impact factors.

jon

+++

Bobby Pickering, Thomson attacks Scopus Citation Tracker feature, Information World Review, February 17, 2006.
[ http://www.iwr.co.uk/2150515 ] http://www.iwr.co.uk/2150515

Thomson has three objections: (1) that Elsevier only tracks citation data back 10 years, so far, (2) that Elsevier focuses on the natural and social sciences instead of covering all disciplines, and (3) that Elsevier moves away from pre-defined
metrics like impact factors and lets users view citation data in many different ways, including citations per article and citations per author.

My take: Thomson is clearly construing every difference from its own model as a weakness. While the first two are real weaknesses, they're also side-effects of the newness of the service and will likely disappear over time. The third is actually
a strength of the Elsevier model. As citation data become available for many different kinds of processing, Thomson has decided to fight a losing battle: defending the impact factor as the single best perspective on the data, even one that is
necessary to make sense of every other perspective.

Peter

+++

Alireza Noruzi, "The Web Impact Factor : a critical review," The Electronic
Library 24 (2006). Self-archived February 9, 2006.
http://eprints.rclis.org/archive/00005543/

Abstract: We analyse the link-based web site impact measure known as the
Web Impact Factor (WIF). It is a quantitative tool for evaluating and
ranking web sites, top-level domains and sub-domains. We also discuss the
WIF's advantages and disadvantages, data collection problems, and validity
and reliability of WIF results. A key to webometric studies has been the
use of large-scale search engines, such as Yahoo and AltaVista that allow
measurements to be made of the total number of pages in a web site and the
total number of backlinks to the web site. These search engines provide
similar possibilities for the investigation of links between web
sites/pages to those provided by the academic journals citation databases
from the Institute of Scientific Information (ISI). But the content of the
Web is not of the same nature and quality as the databases maintained by
the ISI. This paper reviews how the WIF has been developed and applied. It
has been suggested that Web Impact Factors can be calculated as a way of
comparing the attractiveness of web sites or domains on the Web. It is
concluded that, while the WIF is arguably useful for quantitative
intra-country comparison, application beyond this (i.e., to inter-country
assessment) has little value. The paper attempts to make a critical review
over literature on the WIF and associated indicators.

Faked clone research shows chinks in peer review's armor

2005-12-24T22:08:00.000-08:00

Nicholas Wade's article "Clone Scientist Relied on Peers and Korean Pride" in today's New York Times reminds us of some of the ways that closed peer review can be prone to abuse. This analysis asked why the falsified research reported by South
Korean stem-cell researcher Hwang Woo Suk made it through the much-vaunted peer review gauntlet of journals like Science and Nature.

http://www.nytimes.com/2005/12/25/science/25clone.html

Apart from Hwang's apparent knack for pulling the wool over the eyes of his co-workers and funders in the South Korean government, Wade reports that Hwang was adept at attracting co-authors from the biomedical insider's club:

"In addition, Dr. Hwang invited well-known American researchers to be co-authors on his articles, which he may have hoped would make his findings more acceptable to leading journals like Science and Nature. He even invited Dr. Gerald Schatten, a
stem cell expert at the University of Pittsburgh, to be the lead author on the June 2005 report although Dr. Schatten had done none of the experiments....An indication of Dr. Hwang's good connections to the government was the inclusion of Dr. Park
Ky Young as a co-author of his 2004 report on human cloning. A botanist by training, Dr. Park may not have contributed much scientifically to the task of cloning of human cells. She is, however, the science adviser to Roh Moo Hyun, the president of
South Korea."

Curiously, it seems Hwang also exploited a vulnerability in the publication paradigm for experimental science, so often touted as more rigorous than research in the humanities:

A question both journals have considered is that of whether their editors and reviewers should have caught the errors in Dr. Hwang's papers before publication. But as in past cases of fraud, the journals' editors and other scientists assert that
their system depends basically on trust and that reviewers can check only whether a report's conclusions follow from the data presented. "Peer review is not set up to test for fraud," Dr. Campbell said. "It is set up to provide expert assessment of
the scientific credibility and reliability of what scientists report, taking the report itself in good faith." Dr. Kennedy noted that journals often published articles that were later shown to be innocently in error. "The public needs to understand
that the journals and peer review are not perfect," he said.

So where did the truth come out? On the Web, predictably, posted by younger researchers who spent less time accumulating vanity co-authorships and more time analyzing the data:

"It was also South Koreans who took the lead in detecting Dr. Hwang's falsifications. Dr. Zach Hall, president of the California Institute of Regenerative Medicine, noted that young South Korean scientists had brought to light many problems with Dr.
Hwang's papers in Web site postings."

jon

Howard White's co-citation visualizations

2005-12-08T06:35:00.000-08:00

Jim Campbell, a U-Me grad student who I hope will be joining this list soon, recommended this article to me:

Howard D. White, "Pathfinder Networks and Author Cocitation Analysis: A Remapping of Paradigmatic Information Scientists"
http://www.umit.maine.edu/~jon_ippolito/cip/white_pathfinder_networks@m.pdf

It's a bit technical, but I was interested in the social patterns revealed by measuring co-citations--i.e., when researchers cite each other. For example, lots of folks online are going to cite Lawrence Lessig, but only those researchers whom he
also cites are part of his "intellectual circle." Hence, the number of citations to a work from inside or outside a researcher's intellectual circle give a sense of their relevance locally and globally.

jon

Leydesdorff on citation impact visualizations

2005-11-28T21:30:00.000-08:00

Peter Suber brought this article to my attention (and it's available in pdf AND html :)

Although Leydesdorff's approach is limited by his choice to aggregate citations by journals instead of individual articles, I liked his symbolic manipulation of the shape of each node--stretching them vertically or horizontally--to denote additional
information about the journals themselves.

--jon

Leydesdorff, Loet (2005) Visualization of the Citation Impact Environments of Scientific Journals: An online mapping exercise. In Proceedings Annual Meeting of the Society for Social Studies of Science (4S), Pasadena, California.

http://dlist.sir.arizona.edu/992/

Abstract

Aggregated journal-journal citation networks based on the Journal Citation Reports 2004 of the Science Citation Index (5968 journals) and the Social Science Citation Index (1712 journals) are made accessible from the perspective of any of these
journals. A vectorspace model is used for normalization, and the results are brought online at http://www.leydesdorff.net/jcr04 as input-files for the visualization program Pajek. The user is thus able to analyze the citation environment in terms of
links and graphs. Furthermore, the local impact of a journal is defined as its share of the total citations in the specific journals citation environments; the vertical size of the nodes is varied proportionally to this citation impact. The
horizontal size of each node can be used to provide the same information after correction for within-journal (self-)citations. In the citing environment, the equivalents of this measure can be considered as a citation activity index which maps how
the relevant journal environment is perceived by the collective of authors of a given journal. As a policy application, the mechanism of interdisciplinary developments among the sciences is elaborated for the case of nanotechnology journals.

Re: Current Science issue on citation indexing

2005-11-15T13:19:00.000-08:00

Dear OACI list members,

Peter Suber invited me to join your list because of my work with Interarchive, a consortium that explores distributed models of publication and recognition at http://newmedia.umaine.edu/interarchive/. (I'm cross-posting this to the Interarchive
"recognition-metrics" list.) By way of introduction, I'm an artist, curator at the Guggenheim Museum, and teacher in the University of Maine's New Media Program. My work focuses less on traditional scientific scholarship than experimental,
networked, and fluid paradigms of knowledge gathering.

That said, I was grateful for Stevan Harnad's reference to the Current Science special issue on citation indexing (http://www.ias.ac.in/currsci/nov102005/contents.htm). Having skimmed Jacso, Roth, Lewison, Scharnhorst, Lederberg, and Cronin, I was
struck by the schizophrenic character of much of this journal's reporting on citation indexing (apart from a seemingly universal reverence for citation pioneer Eugene Garfield).

For example, Peter Jacso's comparison of Web of Science, Scopus, and Google Scholar, "As We May Search," begins with an example that serves as a scathing indictment of the narrow range of publications indexed by established systems. Jacso is
referring to Vannevar Bush's 1945 paper "As We May Think":

"Bush 60 years ago contemplated - among many other
things - an information workstation, the Memex. A re-
searcher would use it to annotate, organize, link, store,
and retrieve microfilmed documents. He is acknowledged
today as the forefather of the hypertext system, which in
turn, is the backbone of the Internet.

"He outlined his thoughts in an essay published in the
Atlantic Monthly. Maybe because of using a non-
scientific outlet the paper was hardly quoted and cited in
scholarly and professional journals for 30 years." (p. 1537)

Google, of course, knows the impact of this article, as evidenced by its 120,000 returns for "Vannevar Bush" "As We May Think". Yet in his conclusion Jacso excoriates Google Scholar (only a couple months old) for its buggy handling of Boolean
queries and other interface details:

"Riding on the waves of the regular Google software which
is great for processing the unstructured heap of billions of
Web pages, G-S cannot handle even the meticulously
tagged, metadata-enriched few million journal articles
graciously offered to it by many publishers for free." (p. 1546)

I wonder what Jacso would have concluded about the value of the World Wide Web had he launched Mosaic in 1993 and run into a 404 or two.

Similarly, Blaise Cronin in "A Hundred Million Acts of Whimsy?" cites studies in support of citations:

"Michael Kurtz
and colleagues19 at the Harvard-Smithsonian Center for
Astrophysics have used the NASA (National Aeronautics
and Space Administration) Astrophysics Data System to
compare the obsolescence function as measured by 'reads'
of records in the system with the obsolescence function
as measured by citations. Their statistical analyses show
that reads and cites 'fundamentally measure the same thing,
the usefulness of an article'." (p. 1506)

Cronin does conclude with a weak endorsement of citation based on "trust," but he sets himself up for criticism in the body of his article:

"In the course of writing The Hand of Science7, to which I
alluded earlier, I cited, amongst many others, Eugene Garfield
(EG) and Elisabeth Davenport (ED), both of whom I have
known personally for two decades. The former is best de-
scribed as a professional friend (for example, I co-edited
a Festschrift in his honor), the latter as a frequent col-
laborator, co-author and close friend. I did not cite them
because of our social ties, but because their ideas were
relevant to the work in hand. At the same time, the odds
on their being cited by me are increased as a result of the
pre-existing personal connections: I know them and their
publications well; I interact with them, exchanging thoughts
and materials. In the case of the latter we have been active,
and occasionally co-located, collaborators for many years. A
consequence of my citing EG and ED (and others with
whom I am personally acquainted) is that it reduces, po-
tentially, the likelihood of others in the citable author
pool from being selected. All other things (the citable
work's topicality, relevance, currency, etc.) being equal,
strong social ties will presumably trump weak or non-
existent ties. Granovetter25,26 has discussed weak and strong
ties. Call it preferential attachment, a statistical fact of
not only scientific but also social life." (p. 1507)

I bring up the ambivalence expressed by these researchers not to suggest they should straightjacket contradictory findings into a single thesis, but to ask why they flag the importance of knowledge as a social construct only to shy away from using
social metrics for evaluating its producers.

jon

How to join recognition-metrics email discussion list

2005-10-28T12:05:00.000-07:00

To join the interarchive email list, send an email to Majordomo@lists.berkeley.edu with the message, "subscribe recognition-metrics YourEmailAddressHere". After subscribing, post email to this list at: recognition-metrics@lists.berkeley.edu.

To leave the list, follow the instructions above, but change "subscribe" to "unsubscribe".

Email-based blog posting enabled for Recognition Metrics blog

2005-10-27T09:04:00.000-07:00

This blog entry for the Recognition Metrics blog was sent from my email account. To find out how to publish blog entries from your own email accounts, just contact me.

jon

2005-10-18T11:25:00.000-07:00

Welcome to the Recognition Metrics blog! This blog reports news from the Recognition Metrics working group of the Interarchive project, a team of new media artists, curators, publishers, and scholars developing a new paradigm for distributed publishing. The Recognition Metrics subgroup focuses on new ways to visualize and measure the influence networked artists and scholars have on each other.

The Interarchive working group emerged from the 2005 REFRESH! conference held at the Banff New Media Institute and co-sponsored by Leonardo and the Database of Virtual Art.

jon