sweeper

April 13, 2008 By in Uncategorized 35 Comments

Automatically tag your music collection with metadata from Last.fm.

what it is

A while back Last.fm released a command line tool to retrieve metadata for an arbitrary mp3 from their new fingerprint database. I tried it yesterday and it seemed way better than MusicBrainz. So, as a person with a lot of random mp3s, I cooked up a script for retagging entire folders of songs.

Some neat things used in the script:

  • id3lib-ruby for
    handling mp3 tags
  • Text for calculating
    Levenshtein distance to the nearest correct genre name—amatch is a compiled version
    of the same thing, but not Windows-compatible
  • the incredibly comprehensive Last.fm
    API
  • XSD::Mapping for parsing the XML responses (better than
    Hpricot for small, well-formed documents)

A handy feature in the script is the ability to add the top 10 tagged genres to the comment field, so you can use iTunes or Foobar smart playlists for fancier multi-genre sorting. This is similar to lastfmtagger, but not Mac-specific.

demo

Before running sweeper --genre:

$ id3info 1_001.mp3
*** Tag information for 1_001.mp3
*** mp3 info
MPEG1/layer III
Bitrate: 128KBps
Frequency: 44KHz

After:

$ id3info 1_001.mp3
*** Tag information for 1_001.mp3
=== TPE1 (Lead performer(s)/Soloist(s)): Photon Band
=== TIT2 (Title/songname/content description): To Sing For You
=== WORS (Official internet radio station homepage): http://www.last.fm/music/Ph
oton+Band/_/To+Sing+For+You
=== TCON (Content type): Psychadelic
=== COMM (Comments): ()[]: rock, psychedelic, mod, Philly
*** mp3 info
MPEG1/layer III
Bitrate: 128KBps
Frequency: 44KHz

quickstart

Documentation is here, but for OS X:

sudo port install id3lib
sudo gem install sweeper
sweeper --help

Linux is similar to the above, depending on your distribution.

On Windows, you can just download a zipfile from the Rubyforge page and extract sweeper.exe to somewhere in your path.

I expect this to be eventually replaced by an official Last.fm tool, but for now, patches are welcome. It would be especially nice if someone could write a tutorial to help non-Ruby people install the script.

If you are going to contribute some code, grab the SVN checkout from Fauna, since the gem doesn’t ship with the test mp3s.

SVN, I know—how embarrassing!

35 responses

  2. I suppose I’m one of the non-Ruby people. I got Ruby installed well enough on my WinXP machine, but the sweeper install fails on installing amatch due to a mismatch between MS Visual C versions used to make Ruby (version 6) and the version 7 I have installed. I guess I’ll post a solution if I figure it out, short of finding a version 6 to use or re-compiling Ruby under 7.

    Sounds like a neat idea, though. Thanks for making it!

  3. My fault. I didn’t realize the amatch gem used a compiled extension…it’s not Windows compatible. I switched to the Text gem instead which should be cross-platform.

    Try again with version 0.3.

  4. I’m trying to install the sweeper gem on osx and I get this error on building native extensions:

    checking for id3.h... no
You must have id3lib installed in order to use id3lib-ruby.
...
--ruby=/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby

    The thing is, if I run sudo port install id3lib it says I already have it.

  5. Since you’re using the OS X Ruby, and not MacPorts Ruby, you probably have to specify the library location. Something like:

    sudo gem install id3lib-ruby -- --with-id3=/opt/local/

  6. Sorry, I know this sounds so noobish but I’m having a hard time getting this installed and it looks really useful.

    I tried the above, and I got no luck.

    I also tried compiling it manually and I can see it in /usr/local/lib/libid3

    I’m trying sudo gem install id3lib-ruby — –with-id3=/usr/local/

    And no luck with anything I try. Arg :(

  7. I guess try putting the libid3.dylib or whatever it is in /usr/lib, maybe it can find it there. There is also a chance it might work if you copy it to the /ext/ folder of the gem and then manually run ruby extconf.rb and make.

    I don’t use Leopard or the built-in OS X Ruby so I can’t really help you much more; sorry.

  8. Thanks for the change, Evan. However, now I get a buffer-error somewhere. This is kinda spammy, but I printed a backtrace:

    C:\>gem install sweeper --backtrace
ERROR:  While executing gem ... (Zlib::BufError)
    buffer error
C:/ruby/lib/ruby/site_ruby/1.8/rubygems/package.rb:621:in `initialize'
...
C:/ruby/lib/ruby/site_ruby/1.8/rubygems/command.rb:70:in `invoke'
C:/ruby/lib/ruby/site_ruby/1.8/rubygems/command_manager.rb:121:in `process_args'
C:/ruby/lib/ruby/site_ruby/1.8/rubygems/command_manager.rb:92:in `run'
C:/ruby/lib/ruby/site_ruby/1.8/rubygems/gem_runner.rb:30:in `run'
C:/ruby/bin/gem:23

    It installed a number of dependencies successfully, and now just gets the error immediately upon running the install command. Dunno if the trace helps, but thanks for at least trying to support multiple OSs.

  10. doktorjung: Try this. Sorry it’s so difficult… it installs fine for me in Windows on Rubygems version 1.0.1.

    Mike: Thanks for the tips!

  12. doktorjung: I bit the bullet and made a Windows pre-packaged binary. Download a zipfile from here and the included sweeper.exe should work on any system, even without Ruby installed.

    Thanks for your patience.

  15. evan, this might be a stupid question, but I am getting “Skipped (Fingerprint failed)” error message. Is this due to last.fm outage too? I’m asking because I am not sure if the error is in generating fingerprint or matching the fingerprint.

  17. This is what finally worked for me on Leopard with macports:

    sudo port install id3lib
sudo env ARCHFLAGS="-arch i386" gem install id3lib-ruby -- \
  --with-opt-dir=/opt/local/

  18. Man, so much trouble on Leopard.

    Anyway, the Last.fm servers seem to be back up. Sometimes you might still see a skipped error, but there’s not much I can do about that. It only happens for me about 5% of the time. I’m not sure why you’d get nomethoderror; I’ve never seen that.

  19. Hey, Evan. I actually tried the initial suggestion first (with the updated method in Ruby), and it seems to have built fine now. I get a few of the skipped issues you mention, and I can’t see anything in the existing tags that’s much different… Oh well.

    Neat project. Now I just have to work up the courage to run it on everything. A backup’s probably a good idea, heh. Any plans to support WMA, or is that not even possible using Ruby?

  20. Just to add: my skipped error messages are a little different from the one mentioned above:

    Skipped (XMLParserError - syntax error)

    Maybe that’s somehow enlightening. Thanks again.

  25. Hi. Not sure if I’m doing this right but I’m trying to use this after copying sweeper.exe to C:\Windows and running from cmd.exe

    The thing is, sweeper is skipping all the songs – “Fingerprint failed”. Can you point what I’m doing wrong? This looks real interesting to me.

    Thanks!

  26. windows standalone file bug:

    it fails when you try to use it on a drive different from the one you have your user profile on (%userprofile% environment variable) a temp. workaround is:

    mkdir x:\temp

    set userprofile=x:\temp

    now you can run sweeper on x:\ mp3 directories..

    another tip, make a note of what “set userprofile” returns before you change it, so you can change it back instead of rebooting ;)

  28. Using Mac Leopard 10.5.6 on a PowerPC G4 (QuickSilver) here’s what worked:

    sudo port install id3lib
sudo env ARCHFLAGS="" gem install id3lib-ruby -- \
--with-opt-dir=/opt/local/
sudo gem install sweeper

    This is similar to Jack Danger Canty’s great tip above, except it uses ARCHFLAGS=”” which is what works for PowerPC-based Macs under Leopard.

    Thanks for a great utility, Evan.

  29. Well, after getting sweeper to compile and install successfully on Leopard OS X 10.5.6, I’ve been unable to get any mp3 files to work with the Last.fm database. I keep getting “Fingerprint failed” errors with all mp3’s and at all times of the day for the last 24 hours.

    Here’s the output I get from sweeper run on a sample directory:

    Skipped (Fingerprint failed): 01 Give a Little Bit (Studio Version).mp3
Skipped (Fingerprint failed): 02 She's so High.mp3
Skipped (Fingerprint failed): 03 Bohemian Rhapsody.mp3
Skipped (Fingerprint failed): 06 Classical Gas.mp3
Skipped (Fingerprint failed): 06 Take the Long Way Home.mp3
Skipped (Fingerprint failed): Enron-Ron-Ron.mp3

    Evan, do you have any suggestions for me? What part of the code delivers this error message. Is it always the result of downtime at the Last.fm database site, or are there local reasons why my implementation might not work?

    Thank You,

    Bill

  30. Well, I’ve been able to get sweeper installed and running on two macs: My old PowerMac G4 Quicksilver (PowerPC), and a newer Intel iMac.

    I get the “Skipped (Fingerprint failed)” errors repeatedly on the PowerPC machine, but on the Intel Mac sweeper runs like a charm.

    Perhaps sweeper 0.3.2 is not compatible with PowerPC chips?

    What do you think, evan?

    Thanks again,

    Bill

  31. Hi-

    This is a really neat app, tried it with some success on XP. However, I have a question.

    Somehow it does not seem to pick the top rated genre tag for me. For example, when I run it against Killing Joke “Fire Dances” album, it tags Genre as “Alternative”, and then “post-punk, industrial, industrial rock, rock, new wave, metal, alternative rock, 80s, gothic, alternative” in comments. According to the cloud tag data, the top genre title should be “Post-punk”.

    Am I missing something? It would seem the whole point of this is to take the top tag info and use that as the main genre

    DNO

  33. I have been following this promising utility for a year now but have had little success getting the app to run properly under windows (even when using the %userprofile% hack). If I dont get the “fingerprint failed” error i get a “SOAP::mapping::object” error. I have attempted to get this running inder intel OSX 10.5 and Ubuntu 8.10 unsuccessfully. I have found nothing else that does what this program does and i need it to work.

    Does anyone know of a program that does what this does and actually works or can tell me how to get this running on OS 10.5 or windows?

  35. I’m trying to do the install on snow leopard and having trouble with either the id3lib install or the ruby id3lib install.

    I have ran (with results):

    -sudo port install id3lib

--> success


-port installed | grep id3

--> id3lib @3.8.3_2 (active)


sudo env ARCHFLAGS="-arch i386" gem install id3lib-ruby --\
--with-opt-dir=/opt/local/

--> checking for ID3Tag_New() in -lid3... no
--> You must have id3lib installed in order to use id3lib-ruby.
--> *** extconf.rb failed ***

    Extremely frustrating here as it seems like I’m not missing something. I would appreciate any help. Thanks!

    D