Gotcha of the Day: ImageMagick errors when processing large images

I'm at it again, making images for a Google Map custom map using ImageMagick. As with last time, I'm impressed with how effortlessly these tools can make slicing and dicing large images. But, I also ran into a fresh set of issues.

Error while extracting a PNG from a PDF

Kicking off:

  convert -verbose -density 1322.6667 map.pdf map.png

kicks off the following Ghostscript command:

 "gs" -q -dQUIET -dPARANOIDSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=0 \
  "-sDEVICE=pnmraw" -dTextAlphaBits=4 -dGraphicsAlphaBits=4 "-r1322.67x1322.67" \
  "-sOutputFile=/tmp/magick-XXtlfTcB" "-f/tmp/magick-XXmlIqok" "-f/tmp/magick-XXGNxDG3"

And for large values of density were failing with the message:

Error: /ioerror in --showpage--
Operand stack:
   1   true
Execution stack:
   %interp_exit   .runexec2   --nostringval--   --nostringval--   --nostringval--   2   %stopped_push   --nostringval--   --nostringval--   --nostringval--   false   1   %stopped_push   1894   1   3   %oparray_pop   1893   1   3   %oparray_pop   1877   1   3   %oparray_pop   --nostringval--   --nostringval--   2   1   1   --nostringval--   %for_pos_int_continue   --nostringval--   --nostringval--   1777   0   9   %oparray_pop   --nostringval--   --nostringval--
Dictionary stack:
   --dict:1158/1684(ro)(G)--   --dict:1/20(G)--   --dict:82/200(L)--   --dict:82/200(L)--   --dict:108/127(ro)(G)--   --dict:295/300(ro)(G)--   --dict:23/30(L)--   --dict:6/8(L)--   --dict:22/40(L)--
Current allocation mode is local
Last OS error: 28
GPL Ghostscript 9.02: Unrecoverable error, exit code 1
convert: Postscript delegate failed `map.pdf': No such file or directory @ pdf.c/ReadPDFImage/611.

I figured I had to be running out of some resource, but I couldn't tell what. To help with that, I installed strace (sudo yum install strace) and ran the command:

  strace -f -o /tmp/strace.out convert -verbose -density 1322.6667 map.pdf map.png

convert crashes like one would expect. While looking through the tail end of /tmp/strace.out I noticed this line:

24405 write(6, "\377\377...trimmed..."..., 131072) = -1 ENOSPC (No space left on device)

Of course! A df -h confirmed it: I was running out of drive space in /tmp.

The fix for this was especially easy because I'm running on Amazon AWS. In the Amazon AWS console I made a new, 50Gb EBS partition. I then mounted it from within the server. I then created a new temp directory and made use of it by setting the variable TMPDIR.

In other words, I did something like so:

 sudo mount /dev/sdf /mnt/
 sudo mkdir /mnt/ben
 sudo chown ben /mnt/ben
 mkdir /mnn/ben/tmp
 export TMPDIR=/mnt/ben/tmp
 convert -verbose -density 1322.6667 map.pdf map.png

Took a heck of a long time to finish, but finish the command did.

Error while creating tiles

Creating 256x256 tiles using ImageMagick is almost trivial. I just ran:

  convert -verbose -crop 256x256 map.png tiles/t.png

However, for especially large map.png files, I received the message:

24318 Killed                  convert -verbose -crop 256x256 map.png tiles/t.png

I figured it was the system that killed off the process, and so I checked /var/log/messages to see why. Sure enough, there was this line present:

DDD XXX kernel: [35532.036652] Out of memory: Kill process 24318 (convert) score 986 or sacrifice child

Running out of memory definitely made sense. A few options to fixing this included having the system use swap space or upgrading to a larger Amazon instance. Luckily, there's an easier way. Turns out, you can tell convert that it has limits to the amount of resources that it can allocate. Sure enough, changing the above command to one that convinced ImageMagick it had less memory available to it worked like a charm.

convert -limit memory 5GB -limit map 5GB -verbose -crop 256x256 map.png tiles/t.png

The command may have taken longer to complete, but it worked just as smoothly.