Caching iOS updates on a Squid proxy server

Update (22 December 2014): The following instructions have been updated and tested with iOS 8.

Right now, my challenge is upgrading almost 200 iPads to iOS 7 with minimal pain (read: zero device handling). Factor in less-than-ideal Internet bandwidth and Apple’s disinterest in allowing proxies to cache iOS updates, and it’s been a bit of a headache.

First, a word of advice: ask your users not to upgrade when prompted. Do this before Apple release a major update, to buy yourself some time to test it on your network and to check that the update is being cached properly.

Hopefully your iPad fleet is already using your Squid proxy. Ours is configured (via Apple’s Profile Manager) to use a PAC file when it’s on our WiFi network. The PAC file directs all but onsite requests to Squid.

Unfortunately, iOS doesn’t use the proxy for everything; system update authorizations, in particular, don’t get out unless permitted on your firewall. Here’s the relevant rule on our iptables firewall (no_proxy_ok is one of our custom chains, as is tcp_allowed):

-A no_proxy_ok -p tcp -m comment -m tcp -m multiport -d 17.0.0.0/8 -j tcp_allowed --dports 80,443,5223,2195,2196 --comment "allow Apple services (e.g. APNs, updates)"

Mercifully, the update itself is requested via the proxy, but getting it to cache is non-trivial. Obviously max_object_size needs to be big enough to accommodate a 1GB+ file. I went with 2GB:

maximum_object_size 2048000000 bytes

But this wasn’t enough to get the update to cache. A bit of sleuthing led to the first problem: Apple adds HTTP headers like these to its updates, so Squid discards them:

Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache

The workaround is to break HTTP a little by adding this line above any other refresh_pattern entries in your squid.conf:

refresh_pattern -i appldnld\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate

refresh_pattern -i phobos\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate

This forces Squid to treat objects from *.appldnld.apple.com and *.phobos.apple.com as “fresh” (i.e. cacheable) for 90 days (129600 minutes), no matter what appldnld.apple.com and phobos.apple.com say.

Finally, I made sure appldnld.apple.com requests were excluded from Squid’s delay pools and filtering ACLs; you may need to make similar tweaks. I also found that maximum_object_size wasn’t being applied correctly to cache_dir, so I defined it explicitly, i.e.:

cache_dir aufs /var/spool/squid3 256000 128 256 max-size=2048000000

iOS 7 is rolling out smoothly as I type.

24 thoughts on “Caching iOS updates on a Squid proxy server”

  1. Having only 2.5 mbps for 400 users, with Apple’s devices being the most popular ones, this trick has helped us to regain control of our internet connection. We had blocked iTunes completely because we couldn’t get the darn files to stay in cache…

  2. For some reason this was not working on Debian 7.1 with squid 3.1.20
    I added `ignore no-cache` and made it explicit for zip files.
    refresh_pattern -i appldnld\.apple\.com.*\.(zip) 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate ignore-no-cache

    1. Interesting … no-cache headers are client-initiated, so maybe the on-device updater is trying harder to fetch an uncached file now. Thanks for sharing!

  3. Excellent tutorial. I am a school IT Manager also. I have managed to cache apple store updates using your advice.

    However, our main bandwidth nightmare is that our school distributes our teacher’s notes (in ibook and pdf format via iTunesU) and 150+ iPads have to download the same book in a period of few hours. Therefore, I have added another refresh_pattern:

    refresh_pattern -i ([^.]+.|)(phobos).(apple.|)com/.*\.(pdf|ibooks|ipa) 10080 100% 10080 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate;

    However, books are not cached (TCP_MISS/200, as the URL contains a dynamic part (something like http://xxx.phobos.apple.com/…/…/test.ibook?downloadKey=1384882721_3e35e8cd987509c269f6a14089f81a00 )

    I have tried to turn squid3 dynamic content on and off. I have tried to add the dynamic refresh_pattern (/cgi-bin/|\?) 0 0% 0;

    Have you tried anything similar?

    Best regards.

    1. This has probably already been resolved by the person but in case it’s helpful, I noticed the following:

      It appears your regex had “ibooks” instead of “ibook” which you indicated in the URL provided.

  4. 2014/03/03 01:12:57| parse_refreshpattern: Unknown option ‘appldnld\.apple\.com’: ignore-no-store

    Is anyone getting this with the Windows version of 2.7STABLE8

    1. I don’t think it works with App Store downloads because of their URL structure. But OS X Server does cache apps (if you enable this feature).

    1. It should work with transparent proxies, but I haven’t tested it in this scenario. OS X updates are easiest to cache with OS X Server. Enable “Software Update” for operating system updates, and “Caching” for apps (includes both iOS and OS X).

      1. Thanks! Do you know of a way to cache OS X updates with just Squid ? Trying to figure out if possible without buying a OS X server.

  5. Hi Luke,

    Thanks for the information it has been an absolute godsend.

    Just wondering if there have been any changes to this configuration and are you caching apps as well?

    Thanks in advance!

    1. I’ve just done some updated testing on this. Results below. In short, iOS apps can’t be cached by Squid, but macOS apps can be (directives in reply below).

    1. I haven’t looked into Android app caching, but can confirm this doesn’t work with iOS apps — they are retrieved with a unique authorisation key in the query string each time, which makes them uncacheable (unless you’re running a macOS Server with Caching enabled). macOS apps can be cached with a line like this, however:

      refresh_pattern -i osxapps\.itunes\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate ignore-private
      
  6. hello, please can you help me? i can’t send whatsapp msg with iphone across squid (android works well)

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.