Fixing Bibtex Files Made by Mendeley

Mendeley is a popular citation manager but it doesn’t properly export bibtex files. Bibtex files (with a .bib extension) are used to add your references to papers compiled in LaTeX. You can find many reports of these problems, including the Mendeley support page. Some of these problems haven’t been officially fixed for over 6 years! In this post I will share some code that addresses these problems, including:

  • Constraints on what you can write in the “year” field
  • Extra curly braces around titles
  • Incorrect format of “month”
  • Ability to select what types of entries include the URL (instead of only webpages)

Why Am I Using Mendeley?

First, here’s some background on how I came to using Mendeley. I was using RefWorks for citation management during grad school. Eventually, some minor annoyances piled up:

  • The website wasn’t responsive, and it was actually kind of clunky to look up papers and export Bibtex files.
  • The website was behind a subscription wall associated with my institution. This wasn’t a huge problem, but it was inconvenient to need to proxy through a campus server just to access my references.

This year, I started looking for an alternative. I tried out Zotero for a while but found it lacking (perhaps more on that in a future post). I eventually settled on Mendeley for the following reasons:

  1. Aggregate editing. I can select multiple entries to change a single field (very handy for conferences).
  2. Generally effective importing of references directly from the browser window.
  3. A useful app that lets me save a PDF annotated in Adobe reader as a separate file. I find this helpful because I do a lot of paper reading on my tablet.
  4. In-text searching. Which papers mention game theory? It’s easy to look up
  5. I can restrict what folders are backed up online so I don’t use up all of my space (2GB currently).
  6. Custom journal abbreviations.
  7. Automatic generation of bibtex files.

Problems with Bibtex Files

Overall, I think my workflow of citation management improved. However, there was one major issue, and that was how Mendeley creates bibtex files. There are a number of small bugs in how the bibtex files are formatted, and these create problems when building your bibliography in a LaTeX document. Here is a summary of the problems:

  1. Mendeley adds an extra set of curly braces to titles. This means that capitalization is always based on what you enter in Mendeley. It cannot be modified to accommodate the bibliography style. This has been a known issue since 2010.
  2. The year can only be written as 4 digits, which prevents you from writing “submitted”, “in press”, “to appear”, or similar text where it is needed. This has been a known Mendeley issue since 2009.
  3. Special characters (like “{}” or “#” inside of titles, etc) don’t work right for me when I have them escaped. Perhaps this isn’t really a problem because I can just disable the option to escape them, but selecting this option gives me weird output if I try to use curly braces to force capitalization of title characters.
  4. The only options to control what entries display URLs are “All” or “Only for Webpages”. Many of my entries have URLs that I want to keep in my database but I don’t want in the bib entry. BUT, there are non-web page entries where I DO want the bib entry, such as for journal papers that haven’t been published yet. So the options aren’t flexible enough.
  5. Months don’t appear correctly if they written in the database as words (e.g., June). The problem is that curly braces are added around the month (e.g., {jun}), so it gets converted to display as “jun” instead of “Jun.”. One fix is to store months as numbers. but this would mean I have to fix every paper when I add it to my database (since writing months with words is the default).

Correcting Bibtex Files

My initial solution was to wait until I was finished with writing a paper. Then, I would copy the bib entries from my master bibtex file into a separate file. I would fix the entries manually. This worked OK for a couple of short conference papers. However, this wasn’t practical to keep repeating. So I started looking for an automated fix.

Long story short, I wasn’t happy with what I could find. I found some solutions that used the online Mendeley reference format editor (such as for adding “in press”), but I’ve had problems getting the results to show up in bibtex files. So, I decided to work on my own solution.

I wrote some code in C that fixes my database in ~0 seconds (OK, more like 0.05 seconds). This is great because I just double-click on the executable whenever I make a change to my database and then I get a fixed copy of the bibtex file. My database has over 500 references so this function should take a database of thousands of entries and still finish before you switch back to your LaTeX editor.

What does this code do? It reads the file generated by Mendeley, fixes the entries, and writes everything to a new bib-file. It imports the default bib-file named “library.bib” and creates file “library_fixed.bib”, but these can be changed using input arguments. Here’s a list of the exact fixes:

  1. Double curly braces around the titles are changed to single curly braces.
  2. If “issn” is defined for an entry but “year” is not, then the “issn” is renamed to “year”. This let’s you use custom text for “year”, such as “in press”, by omitting the year and writing it under “issn” (my reasoning is that the ISSN is usually not included in a bibliography and it would be highly unusual for a paper to have an ISSN without a year).
  3. Escaped curly braces are changed so that you can use them around specific words in titles and they will display properly (needed for acronyms, proper names, etc.). This is an “exception” to a problem created by the removal of double curly braces for titles.
  4. Curly braces are removed around the month (this will only happen if the month appears in the format {xxx}).
  5. URLs will be removed from all entries except a list of specified types. I currently have “misc” and “unpublished” as default exceptions, and more can be added in the source. Web pages are exported as “misc”.

Your mileage may vary with your own bibtex files, but I’ve posted the code on Github. The windows executable is also posted there if you don’t want to compile it yourself.

I will update the code if I find any bugs or add other changes. If you try it out and find it helpful then I would be happy to know!

17 thoughts on “Fixing Bibtex Files Made by Mendeley

  1. This is the most useful post I’ve recently read about LaTeX & Mendeley. I’ve tried lots of reference managers and I’ve ended up using Mendeley becouse of the features you described. The main advantage for me is renaming fields in scope and the metadata identification when you drag a PDF. Since I’m not a programmer, I have a main doubt about your script: Is it easy to run this C code from mac compiling it myself? and if yes.. Could you give some basic steps please?
    Congratulations for this good job, I’m sure lots of people will take advantage of it.

    Like

    1. Thanks Rafa, I’m, glad you found this useful. Unfortunately, I don’t have much experience with macs, or I would have provided a mac version myself. The code should compile readily with any mac compiler that can work with the C99 standard of C (which should be any modern compiler).

      From a quick web search, you can get the mac version of the gcc compiler by installing XCode.

      You don’t need to open XCode, since you can compile my program directly from a terminal. A command like “gcc -o mendeleyBibFix mendeleyBibFix.c -std=c99” should work.

      I hope this helps!

      Like

  2. Hey,
    thank you for this nice post.

    I have one question though, did it also happen to you that the BibTex file did not include all the documents that you have in your bibiliography?
    I am currently writing in Latex and somehow, some documents are never transferred to the BibTex file, never mind if I create it for the whole library, per group or per document.

    Do you maybe have an idea what the problem is?

    Thank you so much for your help in advance…I am getting a bit depressed here.

    Like

    1. I don’t think that I’ve had that problem. I usually have Mendeley set up to sync Bibtex for my whole library. However, I think that it withholds entries that aren’t “approved”. When you import references, they aren’t assigned a citation key (which Bibtex needs) until you manually confirm that the entry details are correct. I usually see a prompt at the top of the “Document Details” pane (View -> Show Document Details) which asks for confirmation. Perhaps this is your problem?

      Like

      1. Thank you very much for your code! I actually stumbled upon this page while looking for an answer for the exact same problem that some citation just wouldn’t show up in the BibTex at all. Confirming the “Document Details” did indeed the trick 🙂

        Like

  3. Hi there

    How do I permanently delete the .bib file altogether? I want to do in-text referencing(citation) with some of the files in that .bib file, but it keeps showing one article even though there are 30 linked files in there. When I trie to permanently detele so that I can re-add the files individually, I am struggling. It removes it from the library tab, but not from Mendeley. Please help.

    Like

    1. Hi, it sounds like you have 2 separate problems:
      1) Menedeley is auto-creating a bib-file that you want to manage yourself
      2) You’re trying to cite papers in the bib-file but they are not appearing in your paper.

      I will address 1) for now. To disable the auto-create/update of the library bibtex file, go to Tools -> Options -> Bibtex and disable “Enable Bibtex syncing”.

      2) could depend on what software you’re using as an editor/IDE for your latex, or the cite commands may not be entered properly. There are lots of sample templates available online.

      Like

  4. Thank you so much, this tool has saved a huge amount of headache trying to deal with my references for my thesis.

    Thank you!

    Like

  5. Thanks a lot! This has exactly what I was looking for (it’s a shame Mendeley doesn’t update these things fast enough).
    But is it possible to change the set of corrections I would like to apply? I just want to get rid of curly braces around months, annote and file fields, while keeping everything else as it is.
    Thank you in advance! This is really great work 🙂

    Like

    1. I’m glad you found it useful. My short answer is that you could modify the source code and delete/comment the changes you don’t want. That’s not particularly user friendly, but I have no plans at this time to extend this tool to make those kinds of changes through the UI. You could always post a feature request, though 😉

      Like

  6. Hi, Really glad i stumbled upon this -but im having trouble with using the windows exe to add the fix. i’m not sure what to do..
    or do i add the script into the preamble of the .bib file? – is that the same as copy and pasting it at the beginning of the .bib file?

    thanks!

    Like

    1. Thanks for trying out the program. The readme describes how to call the executable, but I didn’t say that you first need to open a command window and navigate to the folder containing both the bib-file and the executable, and then enter “mendeleyBibFix.exe OUTPUT_FILENAME INPUT_FILENAME”, where you specify the input and output filenames. You can omit those filename arguments if your bib-file is called library.bib and you want it to create library_fixed.bib. You don’t need to modify the bib-file directly – that’s what this program is for!

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s