ImageIngester Blog

Use this new version cautiously, as it has a newly re-written ingestion engine.

Updated 5-July-2008 to reflect changes in Version 3.1.04.

I’ve just completed the OS X version (3.1.01) of one of the most significant ImageIngester updates ever. You can now load up to 8 cards at a time and ingest them together, using as many cores/CPUs as your computer has. It’s a winner even with a single core/CPU, since ImageIngester spends most of its time waiting for disk and card I/O (”input/output”) anyway.

For testing I used two powered USB hubs, each with 4 card readers, but any combination of readers will work, with any combination of connection both directly and through hubs. You can use a specialized device like the Delkin ImageRouter, but you don’t have to. (It’s much cheaper to use regular card readers.) I haven’t tested FireWire readers, but they should work OK. (Let me know if they don’t.)

The time to ingest, say, 4 cards at once is often less than 4 times the time to ingest one card because of overlap of CPU and I/O, and because of improved disk scheduling. Sometimes you have a disk bottleneck and there’s no time advantage. Still, it’s really convenient to load all of your cards from a shoot at once and leave the whole lot to ingest while you do other things, rather than having to keep checking so you can load in the next card.

In a test with 4 cards, I got the most favorable times when I ingested into just the primary folder, with no backup, even though that’s not how I’d suggest you use ImageIngester. It took 2:02 (minutes and seconds) to ingest 4 cards, and 1:55 to ingest only one. (Dual-core 1.83 GHz iMac with 1.5 GB of memory and a 500 GB FireWire drive for backup. Rebooted before each ingestion.)

With a pre-conversion backup, but no DNG conversion, there was no time advantage for ingesting multiple cards at all (just the convenience): 9:26 for 4 cards and 2:08 for one. My theory is that, of the 9:26, about 7 minutes were for copying from the backup to the primary folders, and there were only two drives, the external and the internal. So, since the I/O capacity of the drives were fully used, copying more data simply took longer and even created some bottlenecks. Having dual cores didn’t help because the process was entirely I/O bound. For the initial copy from the cards, however, the reads were from four independent card readers, allowing for much more overlap.

(This suggests that a different method for copying to create the pre- and post-conversion backups and the primary copy might make sense when there’s no DNG Conversion, but that’s not something I’m going to explore right away. It’s more important right now to get this new version stabilized and to work on the Windows version.)

For ingestion with DNG Conversion, which is how I use ImageIngester, the times were 14:00 for 4 cards and 4:58 for one. This test had just as much copying as the previous one, without DNG Conversion, but a lot more CPU time, so there was more opportunity for overlap between CPU and I/O.

To use the new multi-card feature, instead of the Start button you press the Cards button:

This brings up the new Ingestion Monitor panel:

From this panel, you see up to 8 mounted cards, and you can press Refresh if the mounts occur when the panel is already up. You can press Eject to eject all the cards, but only after all ingestions are complete. You press Close to close the panel.

You press Start to start the ingestions, and Stop to interrupt processing and stop.

Not all ImageIngester features are available when you’re ingesting from multiple cards, as this chart shows:

[updated]As the chart indicates, I’ve temporarily disabled partitioning, but I expect to have it back soon.

The reason for not providing a way to eject single cards or not ejecting them automatically is that it’s too hard to identify the separate readers. Removing the wrong card could be disastrous! To see the problem, here’s the Ingestion Monitor with 4 Leica M8 and 4 Nikon D200 cards loaded. See if you feel comfortable telling which is which within the Leica and Nikon groups.

(Someday I hope reader manufacturers will provide a way to tell one reader from another, and camera manufacturers will put useful electronic labels on their cards.)

As usual, please give me your feedback either as comments here, in The DAM Forum ImageIngester area, or emailed directly to me.

Version 3.0.02 has just been posted, with these improvements:

• New checkbox to enable metadata; if unchecked, no metadata is generated.
• New Preferences panel to set global options (update check, log, and store ingestion details in database).
• Quick Fields are now sorted in the main window.
• Fixed bug: Some changes failed to set the orange change indicator.
• Choosing Default in the Master preset now shows “Default” as selected.
• Fixed bug on OS X: Was failing to respond to card mounts. May have affected Auto Eject as well.
• On OS X, now shows description of card reader instead of just card label.
• Several other minor problems fixed.

ImageIngester Version 3 has a redesigned user interface that’s simpler, more powerful, more convenient, and less confusing the the one in Versions 1 and 2. The overall design of Version 3 was suggested to me by Peter Krogh, author of The DAM Book: Digital Asset Management for Photographers, whose ideas have greatly contributed to the evolution of ImageIngester from the start. Peter’s ideas were based on his own usage of ImageIngester and his experiences conducting DAM seminars and talking to photographers.

Update: You can now ingest up to 8 cards simultaneously.

Version 3 is a free update for Version 2 users. (There’s no free version.) Mac OS X and Windows betas are available now (on the usual Download page). They seem to work OK, but use them with caution.

Version 3 is intended to address several problems with the older (Version 2) interface:

• Nearly all settings in Version 2 were made on the multiply-tabbed Preferences panel. The panel itself was easy enough to use, but you had to show it to check settings prior to ingestion. The new interface displays all of the important settings on the main window, along with buttons that take you directly to choosing the source (called the “card” in Version 2), setting the destination (dealing with folders), and setting file-handling options (such as file-naming and extensions to be skipped).
• Version 2 had a Preference set that stored all settings, and any changes were automatically saved. This caused simple setting changes that were intended to affect only one ingestion to accidentally apply to subsequent ingestions, even if ImageIngester was relaunched. Version 3 instead has what are called presets, which operate like those in Adobe apps and the Mac print dialog: Settings are saved only if you explicitly save them.
• In addition to a Master preset, which stores almost all settings (analogous to Version 2’s Preference set), Version 3 also has presets for Destination settings, File-Handling settings, Metadata settings, and Keyword settings. So, if you have, say, two typical Destination settings, one for your office and one for the field, you can set them up just once, save them as two Destination presets, and then reapply them as needed for many Master presets. Unlike with Version 2, you don’t have to keep entering the same settings over and over. The Master preset stores the detailed settings, not the name of a Destination, File, Metadata, or Keyword preset, so when you choose a Master preset the names of the other presets are cleared from their drop-downs. This also means that changing a non-Master preset later won’t affect a Master preset unless you explicitly save it again.
• The metadata feature in Version 2 had evolved to a workable design, but too much old stuff was left over. There was no need for the Metadata Template to appear on a main-window drop-down, since the template ImageIngester-IPTC was universal; setting it to anything else usually led to trouble. The IPTC preset panel had 6 tabs, but Peter Krogh suggested that the various fields really fell into just three categories: Photographer, Content, and Rights. So, Version 3 dispenses with the choice of an underlying metadata template and removes the Deprecated tab (which incorporated an old and awkward design). You can now easily set up templates for Photographers, Content, and Rights, and then combine them in various ways that you save as Metadata presets.
• Quick Fields were a handy way to enter information that varied with each ingestion (e.g., subject and location), but they were confusing to set up. You first had to enter a label on the Preferences-Data panel, and then use exactly that same name in a metadata template or in file/folder naming. Version 3 no longer requires you to define Quick Fields in advance of using them (the Data panel is entirely gone). Any appearance of a Quick Field macro (e.g., {@qf.subject}) automatically causes a field by that name to appear on the main window. Any misspellings are immediately obvious.
• The Version 2 ingestion table in the database was valuable, but it didn’t store all of the ingestion settings. In Version 3 these are stored in the database and also are written to the log. A new Details button on the Database panel shows you the settings used for any ingestion. It’s now possible, for example, to discover exactly what Camera Raw settings were used for an ingestion, or what backup folder was used.
• Information to be written to the XMP metadata sidecar is now shown on the Examine panel, as IPTC macros. These are the macros actually used in the ImageIngester-IPTC template, so you can preview how the settings, macros, and metadata templates (Photographer, Content, and Rights) interact prior to ingesting.

Once the settings for an ingestion have been specified, Version 3 ingests exactly as Version 2 did, using exactly the same code. So, there are no new ingestion features and also, hopefully, no new ingestion bugs. Later, after the new user interface has matured and become widely used, new ingestion features will be added to Version 3. I’m chasing one rabbit at a time, as the saying goes.

When Version 3 starts, it converts Version 2 Preference sets to Version 3 Master presets, so in most cases you can use the new version the same way you used the old one. Any changes made in Version 3, however, aren’t copied back to Preference sets.

Since Version 3 is such a big departure from earlier versions, some confusing and little-used features have been removed or changed:

• The Destination folder drop-down on the main window has been removed. Now the folder structure is controlled only by the root folder (primary, pre-conversion backup, or post-conversion backup) and the folder naming. Having a third folder between the root and the folder naming was confusing and of very little value. If you still want to specify a folder at ingestion time, you can use a Quick Field.
• As I mentioned, the choice of an underlying metadata template has been removed, and the entire metadata user interface has been redesigned, although the fields are the same. (A little problem that this introduced is that there’s no way to prevent generation of XMP sidecars. This will be fixed.)
• The starting number on the main window is no longer stored in the Master preset (or any other preset), but it is remembered when ImageIngester is relaunched. Features will be added later to reset it to 1 when ImageIngester is launched or when a new ingestion begins.
• Temporarily, some global settings (log and update check) are always on. A new preferences panel with these will reappear as work on Version 3 progresses.