sweeper

Automatically tag your music collection with metadata from Last.fm.

what it is

A while back Last.fm released a command line tool to retrieve metadata for an arbitrary mp3 from their new fingerprint database. I tried it yesterday and it seemed way better than MusicBrainz. So, as a person with a lot of random mp3s, I cooked up a script for retagging entire folders of songs.

Some neat things used in the script:

  • id3lib-ruby for
    handling mp3 tags
  • Text for calculating
    Levenshtein distance to the nearest correct genre name—amatch is a compiled version
    of the same thing, but not Windows-compatible
  • the incredibly comprehensive Last.fm
    API
  • XSD::Mapping for parsing the XML responses (better than
    Hpricot for small, well-formed documents)

A handy feature in the script is the ability to add the top 10 tagged genres to the comment field, so you can use iTunes or Foobar smart playlists for fancier multi-genre sorting. This is similar to lastfmtagger, but not Mac-specific.

demo

Before running sweeper --genre:

$ id3info 1_001.mp3
*** Tag information for 1_001.mp3
*** mp3 info
MPEG1/layer III
Bitrate: 128KBps
Frequency: 44KHz

After:

$ id3info 1_001.mp3
*** Tag information for 1_001.mp3
=== TPE1 (Lead performer(s)/Soloist(s)): Photon Band
=== TIT2 (Title/songname/content description): To Sing For You
=== WORS (Official internet radio station homepage): http://www.last.fm/music/Ph
oton+Band/_/To+Sing+For+You
=== TCON (Content type): Psychadelic
=== COMM (Comments): ()[]: rock, psychedelic, mod, Philly
*** mp3 info
MPEG1/layer III
Bitrate: 128KBps
Frequency: 44KHz

quickstart

Documentation is here, but for OS X:

sudo port install id3lib
sudo gem install sweeper
sweeper --help

Linux is similar to the above, depending on your distribution.

On Windows, you can just download a zipfile from the Rubyforge page and extract sweeper.exe to somewhere in your path.

I expect this to be eventually replaced by an official Last.fm tool, but for now, patches are welcome. It would be especially nice if someone could write a tutorial to help non-Ruby people install the script.

If you are going to contribute some code, grab the SVN checkout from Fauna, since the gem doesn’t ship with the test mp3s.

SVN, I know—how embarrassing!

This entry was written by evan, posted on April 13, 2008 at 12:00 AM, filed under Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post. Both comments and trackbacks are currently closed.

35 Comments

  1. machinehuman
    Posted April 14, 2008 at 10:42 AM | Permalink

    Nifty. I’ll give it a try. Thanks for sharing.

  2. doktorjung
    Posted April 15, 2008 at 2:26 PM | Permalink

    I suppose I’m one of the non-Ruby people. I got Ruby installed well enough on my WinXP machine, but the sweeper install fails on installing amatch due to a mismatch between MS Visual C versions used to make Ruby (version 6) and the version 7 I have installed. I guess I’ll post a solution if I figure it out, short of finding a version 6 to use or re-compiling Ruby under 7.

    Sounds like a neat idea, though. Thanks for making it!

  3. evan
    Posted April 15, 2008 at 4:46 PM | Permalink

    My fault. I didn’t realize the amatch gem used a compiled extension…it’s not Windows compatible. I switched to the Text gem instead which should be cross-platform.

    Try again with version 0.3.

  4. Daniel Fischer
    Posted April 16, 2008 at 7:05 PM | Permalink

    I’m trying to install the sweeper gem on osx and I get this error on building native extensions:

    checking for id3.h... no
You must have id3lib installed in order to use id3lib-ruby.
...
--ruby=/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby

    The thing is, if I run sudo port install id3lib it says I already have it.

  5. evan
    Posted April 16, 2008 at 7:14 PM | Permalink

    Since you’re using the OS X Ruby, and not MacPorts Ruby, you probably have to specify the library location. Something like:

    sudo gem install id3lib-ruby -- --with-id3=/opt/local/
  6. Daniel Fischer
    Posted April 16, 2008 at 10:01 PM | Permalink

    Sorry, I know this sounds so noobish but I’m having a hard time getting this installed and it looks really useful.

    I tried the above, and I got no luck.

    I also tried compiling it manually and I can see it in /usr/local/lib/libid3

    I’m trying sudo gem install id3lib-ruby — –with-id3=/usr/local/

    And no luck with anything I try. Arg :(

  7. evan
    Posted April 16, 2008 at 10:22 PM | Permalink

    I guess try putting the libid3.dylib or whatever it is in /usr/lib, maybe it can find it there. There is also a chance it might work if you copy it to the /ext/ folder of the gem and then manually run ruby extconf.rb and make.

    I don’t use Leopard or the built-in OS X Ruby so I can’t really help you much more; sorry.

  8. doktorjung
    Posted April 17, 2008 at 1:51 PM | Permalink

    Thanks for the change, Evan. However, now I get a buffer-error somewhere. This is kinda spammy, but I printed a backtrace: 

    C:\>gem install sweeper --backtrace
ERROR:  While executing gem ... (Zlib::BufError)
    buffer error
C:/ruby/lib/ruby/site_ruby/1.8/rubygems/package.rb:621:in `initialize'
...
C:/ruby/lib/ruby/site_ruby/1.8/rubygems/command.rb:70:in `invoke'
C:/ruby/lib/ruby/site_ruby/1.8/rubygems/command_manager.rb:121:in `process_args'
C:/ruby/lib/ruby/site_ruby/1.8/rubygems/command_manager.rb:92:in `run'
C:/ruby/lib/ruby/site_ruby/1.8/rubygems/gem_runner.rb:30:in `run'
C:/ruby/bin/gem:23

    It installed a number of dependencies successfully, and now just gets the error immediately upon running the install command. Dunno if the trace helps, but thanks for at least trying to support multiple OSs.

  9. Mike
    Posted April 17, 2008 at 6:29 PM | Permalink

    I had some trouble getting libid3-ruby installed with the instructions here. I finally had success by downloading libid3 directly and manually installing it, then using the install instructions at libid3-ruby: http://id3lib-ruby.rubyforge.org/doc/files/INSTALL.html

    Thanks for the tool! It’s running now and is looking great!

  10. evan
    Posted April 17, 2008 at 7:12 PM | Permalink

    doktorjung: Try this. Sorry it’s so difficult… it installs fine for me in Windows on Rubygems version 1.0.1.

    Mike: Thanks for the tips!

  11. Seth Thomas Rasmussen
    Posted April 17, 2008 at 7:38 PM | Permalink

    You forgot to mention Choice in your list of neat stuff! I was looking around at ways to do that kind of thing, and I dare say it looks like the nicest choice (I didn’t plan that) I’ve seen so far.

  12. evan
    Posted April 17, 2008 at 11:14 PM | Permalink

    doktorjung: I bit the bullet and made a Windows pre-packaged binary. Download a zipfile from here and the included sweeper.exe should work on any system, even without Ruby installed.

    Thanks for your patience.

  13. Daniel Fischer
    Posted April 18, 2008 at 3:25 AM | Permalink

    Thanks to Mike I figured it out. I had to run this: 

    sudo env ARCHFLAGS="-arch i386" gem install id3lib-ruby

    Then it compiled properly on Leopard. :)

  14. evan
    Posted April 19, 2008 at 9:18 AM | Permalink

    The script is currently returning errors because of the recent Last.fm outage…

  15. chris
    Posted April 20, 2008 at 12:07 AM | Permalink

    evan, this might be a stupid question, but I am getting “Skipped (Fingerprint failed)” error message. Is this due to last.fm outage too? I’m asking because I am not sure if the error is in generating fingerprint or matching the fingerprint.

  16. Daniel Fischer
    Posted April 20, 2008 at 1:53 AM | Permalink

    Ya I get what Chris is getting as well as this: Unknown error: #<nomethoderror:>

  17. Jack Danger Canty
    Posted April 21, 2008 at 10:39 PM | Permalink

    This is what finally worked for me on Leopard with macports:

    sudo port install id3lib
sudo env ARCHFLAGS="-arch i386" gem install id3lib-ruby -- \
  --with-opt-dir=/opt/local/
  18. evan
    Posted April 22, 2008 at 3:39 AM | Permalink

    Man, so much trouble on Leopard.

    Anyway, the Last.fm servers seem to be back up. Sometimes you might still see a skipped error, but there’s not much I can do about that. It only happens for me about 5% of the time. I’m not sure why you’d get nomethoderror; I’ve never seen that.

  19. doktorjung
    Posted April 23, 2008 at 9:20 AM | Permalink

    Hey, Evan. I actually tried the initial suggestion first (with the updated method in Ruby), and it seems to have built fine now. I get a few of the skipped issues you mention, and I can’t see anything in the existing tags that’s much different… Oh well.

    Neat project. Now I just have to work up the courage to run it on everything. A backup’s probably a good idea, heh. Any plans to support WMA, or is that not even possible using Ruby?

  20. doktorjung
    Posted April 23, 2008 at 9:24 AM | Permalink

    Just to add: my skipped error messages are a little different from the one mentioned above:

    Skipped (XMLParserError - syntax error)

    Maybe that’s somehow enlightening. Thanks again.

  21. W. Andrew Loe III
    Posted May 12, 2008 at 7:59 PM | Permalink

    Is there a reason why a lot of my comments are random strings?

    Like for AC/DC – Hells Bells I have:

    00001BE4 00001A51 000058FF 00004AD4 0002E3E0 00021E9F 00008000 00008000 0000EA53 0000B931

    But Last.fm lists some great tags: http://www.last.fm/music/AC%252FDC/_/Hells+Bells

    Suggestions?

  22. W. Andrew Loe III
    Posted May 12, 2008 at 8:14 PM | Permalink

    Bah turns out that I did not pass -f to overwrite my old comments.

  23. Mnky
    Posted June 11, 2008 at 12:05 PM | Permalink

    Does this do album names as well??

  24. evan
    Posted June 11, 2008 at 12:28 PM | Permalink

    Mnky: no, it does not… Last.fm doesn’t provide that data in an accurate way per-track right now.

  25. JonSnow
    Posted July 21, 2008 at 2:35 AM | Permalink

    Hi. Not sure if I’m doing this right but I’m trying to use this after copying sweeper.exe to C:\Windows and running from cmd.exe

    The thing is, sweeper is skipping all the songs – “Fingerprint failed”. Can you point what I’m doing wrong? This looks real interesting to me.

    Thanks!

  26. Pip-Pip
    Posted December 4, 2008 at 10:41 AM | Permalink

    windows standalone file bug:

    it fails when you try to use it on a drive different from the one you have your user profile on (%userprofile% environment variable) a temp. workaround is:

    mkdir x:\temp

    set userprofile=x:\temp

    now you can run sweeper on x:\ mp3 directories..

    another tip, make a note of what “set userprofile” returns before you change it, so you can change it back instead of rebooting ;)

  27. Pip-Pip
    Posted December 4, 2008 at 11:00 AM | Permalink

    As for the “#<nomethoderror:> as they are encountered.. i can send sample files if needed.

  28. macsantos
    Posted December 15, 2008 at 9:48 PM | Permalink

    Using Mac Leopard 10.5.6 on a PowerPC G4 (QuickSilver) here’s what worked: 

    sudo port install id3lib
sudo env ARCHFLAGS="" gem install id3lib-ruby -- \
--with-opt-dir=/opt/local/
sudo gem install sweeper

    This is similar to Jack Danger Canty’s great tip above, except it uses ARCHFLAGS=”" which is what works for PowerPC-based Macs under Leopard.

    Thanks for a great utility, Evan.

  29. macsantos
    Posted December 16, 2008 at 8:53 PM | Permalink

    Well, after getting sweeper to compile and install successfully on Leopard OS X 10.5.6, I’ve been unable to get any mp3 files to work with the Last.fm database. I keep getting “Fingerprint failed” errors with all mp3′s and at all times of the day for the last 24 hours.

    Here’s the output I get from sweeper run on a sample directory: 

    Skipped (Fingerprint failed): 01 Give a Little Bit (Studio Version).mp3
Skipped (Fingerprint failed): 02 She's so High.mp3
Skipped (Fingerprint failed): 03 Bohemian Rhapsody.mp3
Skipped (Fingerprint failed): 06 Classical Gas.mp3
Skipped (Fingerprint failed): 06 Take the Long Way Home.mp3
Skipped (Fingerprint failed): Enron-Ron-Ron.mp3

    Evan, do you have any suggestions for me? What part of the code delivers this error message. Is it always the result of downtime at the Last.fm database site, or are there local reasons why my implementation might not work?

    Thank You,

    Bill

  30. macsantos
    Posted December 30, 2008 at 2:32 PM | Permalink

    Well, I’ve been able to get sweeper installed and running on two macs: My old PowerMac G4 Quicksilver (PowerPC), and a newer Intel iMac.

    I get the “Skipped (Fingerprint failed)” errors repeatedly on the PowerPC machine, but on the Intel Mac sweeper runs like a charm.

    Perhaps sweeper 0.3.2 is not compatible with PowerPC chips?

    What do you think, evan?

    Thanks again,

    Bill

  31. DNO
    Posted December 30, 2008 at 5:10 PM | Permalink

    Hi-

    This is a really neat app, tried it with some success on XP. However, I have a question.

    Somehow it does not seem to pick the top rated genre tag for me. For example, when I run it against Killing Joke “Fire Dances” album, it tags Genre as “Alternative”, and then “post-punk, industrial, industrial rock, rock, new wave, metal, alternative rock, 80s, gothic, alternative” in comments. According to the cloud tag data, the top genre title should be “Post-punk”.

    Am I missing something? It would seem the whole point of this is to take the top tag info and use that as the main genre

    DNO

  32. Henrik
    Posted May 25, 2009 at 5:03 AM | Permalink

    Do you plan to support FLAC files? That would be awesome :)

  33. Paul
    Posted June 1, 2009 at 7:56 PM | Permalink

    I have been following this promising utility for a year now but have had little success getting the app to run properly under windows (even when using the %userprofile% hack). If I dont get the “fingerprint failed” error i get a “SOAP::mapping::object” error. I have attempted to get this running inder intel OSX 10.5 and Ubuntu 8.10 unsuccessfully. I have found nothing else that does what this program does and i need it to work.

    Does anyone know of a program that does what this does and actually works or can tell me how to get this running on OS 10.5 or windows?

  34. Dan McGrady
    Posted July 7, 2009 at 7:08 AM | Permalink

    I had to add require ‘Tempfile’ to sweeper.rb to get this to work (using Leopard/1.8.6).

  35. Daniel Demole
    Posted January 9, 2010 at 9:44 AM | Permalink

    I’m trying to do the install on snow leopard and having trouble with either the id3lib install or the ruby id3lib install.

    I have ran (with results): 

    -sudo port install id3lib

--> success

-port installed | grep id3

--> id3lib @3.8.3_2 (active)

sudo env ARCHFLAGS="-arch i386" gem install id3lib-ruby --\
--with-opt-dir=/opt/local/

--> checking for ID3Tag_New() in -lid3... no
--> You must have id3lib installed in order to use id3lib-ruby.
--> *** extconf.rb failed ***

    Extremely frustrating here as it seems like I’m not missing something. I would appreciate any help. Thanks!

    D

Follow

Get every new post delivered to your Inbox.

Join 66 other followers