Data citation in the wild
Valerie Enriquez, Sarah Walker Judson, Nicholas M. Weber, Suzie Allard, Robert B. Cook, Heather A. Piwowar, Robert J. Sandusky, Todd J. Vision, Bruce Wilson

Consistent attribution of research data upon reuse is necessary to reward the original data-producing investigators, reconstruct provenance, and inform data sharing policies, tool requirements, and funding decisions.  Unfortunately, norms for data attribution are varied and often weak.  As part of the DataONE 2010 summer internship program, three interns studied the policies, practice, and implications of current data attribution behavior in the environmental sciences.  We found that few policies recommend robust data citation practices: in our preliminary evaluation, only one-third of repositories (n=26), 6% of journals (n=307), and 1 of 53 funders suggested a best practice for data citation.  We manually reviewed 500 papers published between 2000 and 2010 across six journals; of the 198 papers that reused datasets, only 14% reported a unique dataset identifier in their dataset attribution, and a partially-overlapping 12% mentioned the author name and repository name.  Few citations to datasets themselves were made in the article references section.  In multivariate analysis, citation patterns were more correlated with repository (with citations to Genbank being most complete) than journal or datatype. Attribution patterns were found to be steady over time.  Consistent with these findings, dataset reuse was difficult to track through standard retrieval resources.  Searching by repository name retrieved many instances of data submission rather than data reuse, combing the citation history of data creation articles was time consuming, and searching citation databases for the few early-adopter dataset DOIs and HDLs in reference lists failed due to apparent limitations in database query capabilities and structured extraction of DOIs.  We hope these descriptions of the current data attribution environment will highlight outstanding issues and motivate change in policy, tools, and practice.  This research was done as open science (http://openwetware.org/wiki/DataONE:Notebook/Summer_2010):  ask us about it!

What then? There is always interlibrary loan. However, there may be cost prohibitions, or the article may not get to you in time.

What if you could contact other researchers in similar fields to obtain access and recommendations to articles?

Shameless plug time.

Heather had introduced me to Mendeley as a way to manage our ever-growing OpenWetware page of web resources relating to data sharing, data reuse and data citation. I also used it to tag my citations for articles I had found citing TreeBASE, Pangaea and ORNL DAAC.

Most importantly, Mendeley allows for the sharing of lists with other users as Public Collections as well as Shared Collections that allow the sharing of documents with up to ten people for reading and tagging. Mendeley also has a plugin for Word and OpenOffice for easy citation.

Here is our current list of Web Resources.

In a way, I suppose I have followed both approaches. I had written out a draft before I even had real results, or even before I had collected all data. On the other hand, here I am, reading more articles for structural ideas, emailing others for their input and additional information, and poring over my data to make sure I didn’t miss anything. I hear this song in my head when I wake up in the morning and fall asleep to the sound of a clock ticking at night.

What does it all mean? Just because I may have a solid article draft to shop around to different publications by Friday or next Friday (or one of the subsequent Fridays after that), does it mean that I’m “done?” This isn’t like turning in an assignment for the final result to show up in final grades at the end of semester. After all, I do have that pesky tendency to raise more questions than I can answer at once, sort of like my tendency to not be able to pick between dessert options, resulting in double dessert Thursdays. Yet each question, like each dessert is a different flavor to satisfy different needs. While double dessert may be plausible, triple or quadruple would be a terrible idea.

The current shape of my questions has changed as much as the focus of my to-be-completed article. “How are data housed in different repositories cited and attributed” eventually became “Why is it difficult to find citations and attributions for reused repository datasets?” I went from originally focusing on a quantitative study to writing a perspectives piece. It was as if I wanted to make a tres leches cake, but only had eggs and condensed milk, so I ended up making a flan instead. While delicious and satisfying (and hopefully informative and potentially provocative in the case of my article), it still wasn’t quite what I originally had in mind.

Yet the wonderful thing about science (and cooking) is that there is always room and time for re-evaluation and innovation. I don’t have to have all of the answers at once, and frankly, if I did, that would hardly be credible. This is what I liked about the idea of data repositories in the first place: the idea that all the hours of work that go into collecting data aren’t brushed aside once an initial product has been created (or in some cases, if a product is not created). The last thing I would want is for my data or my ideas to collect dust on a shelf. What if I could come back to it later with a fresh brain and set of eyes with the 20/20 vision afforded to those looking at things in the right amount of hindsight? What if it could help someone else? What if somebody could use it as a base to build off of for future studies to answer questions I either raised, but could not answer at the time or could not even think of raising? It would be like that layered pecan-sweet potato pie I saw once (perhaps in a dream).

However, I do have to have some answers right now, so it’s great that I have a team to work with who’ll tell me if I’m making wild conjecture or if I otherwise don’t have enough data to back up a claim. Either way, the clock is ticking and the oven is preheated. Let’s make some dessert do some science.

Evolutionary Biology http://tables.googlelabs.com/DataSource?snapid=68246

Ecology http://tables.googlelabs.com/DataSource?snapid=68743

And Envrionmental Sciences http://tables.googlelabs.com/DataSource?snapid=68245

The categories were determined by Reuters Journal Citation Report . I evaluated each journal in these three categories hoping to get a good range of the policies affecting a researcher attempting to publish in this domain.

I’ve also spent a fair amount of time looking into data sharing and citing policies for funding agencies and repositories as well. Our investigations into these sources was an attempt to be holistic, and get a wider view of the policies affecting not just publication but funding and archiving as well.

It’s been an enlightening look at what is surely an emerging area of important for policy development. The variation in requirements and mandates is quite surprising; some agencies say nothing or just a paragraph about the importance of data sharing while others have 15 page manuals with extended commentary on not only the importance of sharing a dataset but extensive recommendations for archiving a number of data types.

It’s also been a frustrating experience in some respects. Some journals with mission statements that stress their devotion to expanding communication in the fields of ecology make no mention of the need to share or attribute data. And not to bite the hand that feeds, but in their grant conditions, NSF devotes as much space to their direction on SEATBELT SAFETY as they do data sharing (www.nsf.gov/pubs/gc1/jan09.pdf).

Nevertheless, I am sure there are portions of my own extraction effort that have either missed an essential policy element or neglected a set of funding agencies important to this specific domain. So I’d love for a community of stakeholders to take a look at my data so far and give me feedback. You can let me know if you see a resource missing,or a mistake by using the comment feature on Google FusionTables (user: nicholas.m.weber) , write on my OWW Notebook Page – or send me an email: nmweber@illinois.edu.

And of course don’t forget to Buckle Up if you’re driving.  (I want to be clear, I’m not belittling safety on the road. Wear your seatbelt.)

Show me the Data: NISo is hosting a webinar on linking datasets and publications  August 11^th

There is a trove of session summaries and interviews with presenters from the CNI / JISC  2010 Meeting : Managing Data in Difficult Times: policies, strategies, technologies and infrastructure to manage research and teaching data in a fast changing technological and economic environment.

(thanks to @simonhodson99)

There is an mp3 and slides available from a dean’s lecture series at UC Berkley’s iSchool by Victoria Stodden. The focus of her lecture is the degradation of the scientific method (namely reproducible findings) in a digital age. Stodden is a fierce advocate of not only open data, but open code ( an important distinction that I think sometimes get’s washed out in the clamoring for making underlying data more accessible) . What’s especially effective about Stodden’s approach, besides its breadth, is her licensing ideas on how we return to sound scientific practice. In gist, her thesis includes  “open licensing options designed to create an intellectual property framework for scientists consonant with longstanding scientific norms” and the establishment of what she calls ‘reproducible research standards’

[via @oatp]

At a basic level, I simply wish authors were more clear (it sure would make my data extraction easier!). This requires transparency on the part of the scientist. Not just a willingness to be open, but explicitly clear articulation of what data is available. And on top of that, where it can be found.  There is a general assumption in the scientific community that if someone really wants the data, they will contact the original author. Authors commonly include this caveat, especially when their data is not posted elsewhere.

But the fact of the matter, accumulating data from multifarious sources is a difficult process. Personally, I have spent many hours trying to track down GIS layers by emailing the original authors or digging through web archives. Given, I was working in a relatively understudied area, but my literature searches produced a fair number of potential sources and most lead to dead ends…non-existent urls, out dated email addresses, etc. Even when working within my established group of collaborators (not DataONE of course!), it could take me weeks of emails and reminders to get my hands on a simple spreadsheet. If I couldn’t get an email response and accompanying dataset from a colleague, what are my chances of getting it from a scientist I’ve never met? Therefore, it’s nice if an author handwavingly says they’ll provide their data, but its much better if they post it, document it, and state in their publication that they did so.

With that said, there are a number of better practices that can be implemented on the journal, editor, and author level to facilitate less cumbersome data sharing. Here are some of my preliminary suggestions on good citation practices that enable data to be traceable and truely reusable:

• Accession numbers and Authors of each dataset (reused and shared) given in the Methods

Alternatively, for large datasets posted in multiple places, a Table or Appendix referenced in the Methods should be given

Authors should be not charged extra page fees for including all Bibliographic citations for all original data authors or a table of relevant accession numbers and reference

• Proposed Bibliographic Citation Format:

Author. Year. Article Title. Journal. Pages. Depository. Accession Number.

This is probably old news in this field, but this format seems most intuitive to me and would have been helpful in my data extraction. I think it would be useful to lump the original article and the dataset since most reuse instances typically give the Author Year citation when they are really referring to the dataset, but at the same time the article is needed for context of the dataset. Also, this would also authors to track their dataset reuse through ISI and other aggregators.

• Editorial enforcement

I envision this as a simple checklist for the copy editors (not reviewers) to confirm that accession numbers and/or an author reference are given for all reused and shared datasets.

As an initial step, this is most easily done for GenBank and Treebase which have clear cut dataset types (respectively sequences and alignments/trees). Especially in journals like Molecular Ecology and Systematic Biology, nearly every paper should have at least one of these citations.

• Internal (Journal) Depositories made more accessible

Data should be made available in usable formats, not just static pdfs.

Data should have a unique and stable URL. This is especially a problem in Systematic Biology where all data is said to be stored at http://sysbio.oxfordjournals.org/ which is not unique for an article. Furthermore, data previous to 2008 cannot be found at this site (which is referenced in all pre-2008 articles), but can be found at http://systbio.org  under Appendices and data. Both entry points require searching for the data rather than a direct connection such as the urls provided by Ecological Archives.

• Separate Supplementary Data Section –

In general, there is confusion among authors about what “Supplementary data” is. Often, it is treated as a data dump for extra figures and statistical outputs, not raw data.

Journals could and should have a separate section for accompanying raw data.

For example: Molecular Ecology has a Supplementary data section at the end of each article which has a one sentence description for what is contained in each appendix.   American Naturalist recently added an “Online enhancements” header at the top of each article which often provides links to shared datasets. Systematic Biology has a separate section at the end of the article which is a good first step, but as it is now just contains the same blanket statement along the lines of: ‘data is available at the SysBio website…’

I propose that this additional section contain brief summary info and URL/accession for shared datasets, as well as reused data.

For some context: At our mentor/intern meetup this week, we presented our initial findings as well as some challenges. I had a draft of an article (more like the most basic ideas of an article) and many in the group suggested I rewrite the article in a more narrative tone as an opinions/perspectives piece. Most of my friends would agree that I don’t have a shortage of opinions. However, anything anecdotal is not real scientific evidence (at least not what I’ve gathered from the various scientific blogs I’ve read).

This is where my data comes in. Along with this piece, there may also be a poster for future presentations. Yet I have to wonder how much of this really is just my opinion? I am sort of curious about doing a survey of researchers and information professionals to see if they have also had difficulty finding data citations (not to mention see if there are any knowledge gaps between researchers and information professionals regarding this type of search). Granted, my project was just a general search for data citations across a narrow sample. However, if a general search truly is difficult, then how hard is it to find a specific set of data given a limited amount of known information about the data set (like my previous analogy where finding a data citation by author, repository or accession number was sort of like trying to find someone on Facebook by using their hair color and favorite breakfast cereal)?

I’ve always liked telling a good story. However, in this case, the story must be accurate as well. For my argument to work, there should be a truly demonstrated need for this information to be found easily. However, what is the awareness of this need? How many researchers look for the data that others create? Or, do scientists prefer to run their own experiments/make their own observations? How helpful would it be for them to compare or contrast their data to the data of others? A question unasked is a question unanswered.

Does anyone have thoughts about this? Once again, I’m going back to the ideas of crowdsourcing and open science where people create and share data throughout the scientific research process. On that note, Heather has forwarded this link to me and the other interns: http://www.science2point0.com/ I rather like this idea, a place where the scientific community can share, discuss and solicit requests for information or help with projects.

For now, I’m on my way back to Boston to continue working, continue drafting, continue trying to help figure out better ways to document and share science.

This has been my clumsy way of explaining to my friends exactly what it is I’ve been doing lately and why I can’t come out and play in the tree fort with the other kids.

Not to say that work can’t also be fun. If this project didn’t sound fun, I probably wouldn’t have signed up for it in the first place. Being curious about everything drew me to being a science hobbyist, writer, amateur journalist and eventually an Archives grad student. I was the kid who asked the tour guide obnoxious questions on the field trip when all anyone wanted to do was nap on the bus, read a book or watch a movie during class trips.

Exactly why is it so hard to find citations of reused scientific data? What is an example of a “good” citation of repository data? So far I’m finding that at least one “good” attribute is being unique and easy to find. Citing by author and year can bring up different articles by that author in the same year or even the wrong author if only found via last name+et al. DOIs are proving their value as far as retrievability is concerned, but when half of the findings are “data for this project is deposited/is accessible at at doi: 10..XX.XXX.etc.,” how do we exclude those from the reuse citations? While repositories have their own recommendations: TreeBASE has recommend citations on each page that includes data author and study title (example here); Pangaea recommends including the author, year, title, institution, and DOI;  ORNL DAAC has a similar policy with its own formatting rules. This doesn’t even go into the different recommendation each journal/publication has for citing data, which is another story that Nic’s covering.

It also seems that every question has several other questions that arise with it. If I ask “What is the most common/best way to cite data?” then I also have to consider “Do the best practices/recommendations of each repository actually contribute to making the dataset easy to track and find across other articles?” How much narrower of a focus do I gain by using boolean operators and controlled vocabulary? How much do I risk losing by using such narrow searches? For every search I construct, I feel like I’m missing five others. As a visual person, I even created a table to help me keep track as I went through my earlier journal entries. While I might not have all possible leads, I have a records of the ones I previously tracked. Clues lead to clues. Questions lead to more questions. Eventually there are answers… or at least, I hope there will be answers.

That may have just been the wannabe gumshoe talking in me: the kid who always watched Where in the World is Carmen Sandiego on PBS and played all the games, the teenager who wrote amateur detective stories and loved Patricia Highsmith, Dashiell Hammett and film noir a bit too much. I’d put my trenchcoat and fedora on if it weren’t so darn hot outside.

To follow Sarah’s post, I agree that the dead ends are just as important as the big leads. Anyone who’s ever gotten lost in an unfamiliar town knows that they have to get stuck in a few places and pull over into at least one gas station before they get back to the highway. Of course, now there’s GPS and the like, but there aren’t always shortcuts like that in finding information. Will DOIs be the GPS for data citation? So far, I’m finding that they have a high hit count compared to the other methods of sampling I’ve used (author name or repository name).

Back to my opening analogy,  DOIs have a lot more structure and standards than something as arbitrary as hair color or favorite breakfast cereal. However, they’re not always used in citation. I’ve found articles that either only mention the author or the name of the repository from which the data came. Yet with all the variation in names (last name only, last name and first initial, full name last and first, etc.), it might as well be like trying to find someone on Facebook or MySpace by breakfast cereal name (Cap’n Crunch? Puffins? Sweetened puffed corn rectangle pillows? Peanut butter puffs?). I feel strongly that if people won’t cite names consistently, the least they could do is use the DOI, which is more or less the equivalent of finding someone by their phone number or email address or class schedule on Facebook (except more helpful and not creepy).

I feel like the amount of attention I’ve been paying to small details like search strings is just part of an even greater picture outside of data reuse. I’m sure that once Nic, Sarah and I combine and compare data, we’ll have a better look at that bigger picture. Perhaps it won’t be the whole picture, but it’ll certainly be more than the random handful of puzzle pieces I have in hand now.

And thus, scientists go about mining data. Pilfering published articles, scanning internet resources, pestering their colleagues for dormant data. At times there is a preponderance of data, but it is un-standardized and therefore unusable without performing an obstacle course of statistics and normalizations. Other times, there is no accessible data even though numerous publications allude to existing data. So, what is a meta-analyst to do?

Well, it is our hope in this project that we can establish good practices of data reuse and sharing: good metadata for data reusability, common depositories for data access, proper citation of data to give credit where it is due. Also, it is our long term hope that data will be cited and recognized on the same level of publications, both for the benefit of the meta-analyst seeking data and the professor seeking tenure, not to mention the amateur and citizen scientists seeking to solve local issues.

Now, let me step back from the philosophical benefits of data reuse and recount some examples I have encountered that illustrate the need for good practices (which inspired this post in the first place). Some of the very first articles I analyzed aptly illustrated the shortcomings in current data reuse, but also the sustained interest in utilizing such a vast resource. A number of articles haphazardly credited data sources and more predominantly cited their own previous data at the expense of other sources. There are too many of these situations to count. Overall, there is a lack of consistency in how to cite the dataset and make it retraceable for future use.

A more concrete and poignant example is an instance of an author seeking to utilize Treebase (http://www.treebase.org) datasets. In order to test a proposed method, the author sought the original data matrices of a number of phylogenetic studies. In doing so, the author provided a mini commentary on the arduous search-and-seive process of finding quality data. First, effective keywords had to be ascertained. This is essential to all literature searches, but as Valerie well knows, it is even more critical in elucidating datasets. Then, the author sifted through the initial results, only to find that many files posted on Treebase lacked necessary metadata that precluded re-analysis. After examination of the available data, the author settled on a pitifully few case studies to illustrate the proposed method. Ironically, after all the trouble he went through, the author did not provide references or accession numbers of the datasets utilized, nor did he share the compiled dataset.

Often, authors are discouraged from articulating the “dead-ends” in their research as it distracts from the intent of the paper. But in this case, and for my purposes, this was the most informative part of the manuscript. It illustrates the struggle of finding, accessing, and sharing data. There is interest in reusing data and this author expressed frustration that it was not more accessible, yet it feels like there is so far to go. At some level, despite the efforts of depositories, journals, and entities like DataONE, it is ultimately the job of each scientist to take responsibility for this desire by posting their data, utilizing and crediting other data, and being transparent in this process. Though this author lamented the sad state of Treebase reuse, he did not also do a “good turn” and reciprocally share his data or explicitly articulate data sources. I believe that the best remedy to the demand for data is openness and sharing on the part of individual scientists.

Luckily, examples of this exist as well. Two articles which I recently read described the difficult process of extracting various parameters from 50+ publications. I sympathized from my experience performing the same process. In both articles, it at first seemed as though the authors were detailing their extraction methods in such a way that would encourage others to perform a similar process. As I read through the lengthy methods, I fully anticipated that since they covered their methodology so thoroughly, they surely wouldn’t post the actual dataset. Yet, in both instances, I was delighted when a simple sentence indicated that the data was available online. I was even more pleased when I downloaded the files, discovering that they were in a useable format that also indicated the original manuscripts. The detailed methods and crossreferences were the necessary metadata for replication or re-analysis, and the raw data now sits comfortably on my personal desktop (should I ever decide to investigate the ecology of orchids or bats).

So, there is hope. And much of it stems from the initiative of scientists. Though I have not quantitatively accessed trends as of yet, it seems that many instances of reuse or sharing occur for personal, unarticulated reasons. Perhaps, a scientist benefitted from data sharing in the past or simply believes in the transparency of science. Conversely, they may have had their data preemptively published by a greedy colleague and therefore hoard their data. On both sides of the fence, there are risks and benefits. Regardless of individual experience, with each publication there are more calls for data to answer deeper and broader questions. I have hope that scientists will rise to the challenge of meeting this demand, and that our work can aid them in navigating the jungle of raw data that awaits.

It’s hard for me to imagine a time before the internet. Then again, what I have pictured usually involves Thomas Edison being a jerk and trying to vandalize Nicola Tesla’s OpenWetware pages. With the rise of Open Notebook Science does this mean that the days of academic rivalry are done? I would like to think that the new openness encourages collaboration as opposed to competition. I like checking my talk pages to see if anyone’s left any comments or suggestions, considering how much of a n00b I am at this sort of thing.

Ok, this is veering on the edge of my insecure middle school ramblings and is thus not relevant to science.

I’ve only kept this notebook for about two weeks and it’s interesting to see how notes and casual observations I had made are slowly coming together to form solid ideas about how scientific data is re-used and why it’s so gosh-darn hard to track. Granted, some of my search strings are probably laughable to the more seasoned informaticians out there, but they’re growing and changing every day.

Now that I’m in the collection and analysis stage, what do I do with all these searches (and lists of results for that matter)? I consulted the chapter on Bibliometrics in Practical Research Methods for Librarians and Information Professionals (ISBN: 978-1-55570-591-6) and found that there are various effects and biases I should look out for when interpreting what I’ve found. Some interpretations can be problematic in that they may include presumptions that may have alternate explanations or factors outside of what I have observed. For one thing, is finding data reuse citations really difficult or am I just doing my searches wrong? A look in my notebook entries will show the level of thoroughness for each search and lengths I went through to avoid exclusion with the risk of false drops. Yet was I thorough enough? Are the search results large enough sample sizes?

Yet that is the wonderful thing about open science and science in general. Anyone else could step up to the plate to either help support my claims or completely refute them with new evidence. Or, as my fellow interns and mentors have done, provide commentary and suggestions throughout my research process.

My methodology for finding citations for three repositories (TreeBASE, Pangaea and ORNL DAAC) tended to go from extremely broad (basic keyword search for repository names mentioned in articles) to extremely narrow (names of individual data authors or particular DOIs or study accession numbers) in case I missed something important. While I couldn’t exactly read the fulltext of every search that pulled 1000 articles for content in two weeks, I could run word searches or skim abstracts for any mention of repositories or data authors. So far, I have been more successful in finding citations for individual data authors as opposed to general mentions of data repositories. Perhaps this is because many datasets supplement studies or articles. Or, maybe I’m getting ahead of myself and making the sort of assumptions I’ve been warned against making. For one thing, I readily admit a slight bias against Google; lo and behold, some of my “worst/least helpful” search results have come from Google Scholar.

Image Courtesy of: Respectful Insolence @ Science Blogs

I could be doing things wrong or really wrong, but fortunately, I can get called out for it.  On another note, the design of ISI Web of Science’s Cited Reference Search is aimed more toward finding individual articles or article authors as opposed to datasets. Is it possible that I am demonstrating a potential need for tracking? I don’t imagine I’m the only person out there checking to see how often datasets are (re)used. If scientists and researchers find it useful to track how often their articles are cited, I imagine it could only help the flow of information if data citations are also tracked. For one thing, this sort of tracking could help bring more prestige to the idea of  datasets as valuable publications, not to mention bring prestige to the scientists and researchers themselves.

Soon I’ll be quantifying all of this beyond “this was hard to find” and “some were found this way, but more were found this way.” I’ll also have to keep in mind the basic statistical mantra of “correlation does not necessarily mean causation” while writing my interpretations and conclusions. I know there’s certainly a lot more to this than can be covered in a two-month project, so I can’t wait to see what develops.