Determining Multicast ASR Data Rates

* Introduction

Mac OS X Tiger introduces a terrific new feature in Apple Software Restore: Multicast server and client support. "Multicast" is a data transfer technology in which one stream of packets is sent to one "Multicast network address" that can be received by any number of clients. Contrast this to the previous method of deployment (unicast) in which X number of machines would connect to one server, resulting in X number of streams of the same data. Multicast ASR will allow you to perform your network deployments faster and at a lower cost of network bandwidth.

Among the features of multicast ASR:

  • The multicast client can handle up to 20% packet loss
  • The multicast client can make an out-of-band request to the server for up to 32MB of missing blocks at the end of the first pass
  • The multicast server loops, so the client can retrieve lost packets on a subsequent loop
  • The multicast server and client advertise their state (not progress) via multicast DNS (Bonjour)
  • The multicast server does not begin sending packets until a client initiates communication
  • The multicast server can stop streaming if there are no clients receiving data
  • The multicast server can throttle output based on a configured maximum Client Data Rate

* Running a Multicast ASR server

To initiate a multicast ASR server, read the asr man page and follow steps similar to the following:

  1. Create a master disk image (e.g. using Disk Utility), store it anywhere on a Mac OS X Server
  2. In the Terminal, run the following commands to create a Multicast ASR server configuration file (see configuration tips below):

    defaults write /Users/admin/Desktop/asr "Data Rate" -int 6000000
    defaults write /Users/admin/Desktop/asr "Multicast Address" <mcast address>

    <mcast address> should be appropriate for your network infrastructure and policy, usually from a range assigned by your network administrator. More info below on sane values for multicast address.

  3. Then run this command to initiate the multicast ASR server:

    sudo asr -source /path/to/master.dmg -server /Users/admin/Desktop/asr.plist

  4. The server will wait for the first client to "report in". Once that has occurred, the server will begin multicasting packets from the disk image, looping continuously.
  5. When all of your machines have restored themselves, stop the multicast ASR server by pressing "Control+C" in the Terminal.

* Notable configuration options

There are a few configurable options for the multicast ASR server. Modified from the asr man page:

  • Data Rate
    This is the desired data rate in bytes per second. On average, the stream will go slightly slower than this speed, but will never exceed it.

    Note: The performance/reliability of the networking infrastructure being multicast on is an important factor in determining what Data Rate can be supported. Excessive/bursty packet loss for a given Data Rate could be due to an inability of the server/client to be able to send/receive multicast data at that rate, but it's equally important to verify that the network infrastructure can support multicasting at the requested rate.

    Sane values for Data Rate range from 2Mb/s (2,000,000 -- no commas when setting these values) to 15Mb/s (15,000,000). Data Rate depends on your ethernet bandwidth (10Mb/100Mb/1000Mb) and the capacity of your clients to write data to their hard drives. 12,000,000 is a good value for desktop machines on a 100Mb network, though you can experiment with this value to optimize speed against dropped packets. In my tests on an isolated network, 15,000,000 worked for me on a Dual 2.0GHz G5 with a compressed disk image. If your machines are G4s or G5s, you should definitely use compressed disk images as it typically cuts the restore time in half.

    Consult with your network administrator to determine the best Data Rate to use. Also, the presence of slower hardware on your network (e.g. old 10Mb printers or PCs) can dramatically reduce the output of the multicast server. If you experience severe packet loss at seemingly low data rates, examine your network to determine if slower hardware on the network could be slowing down the Data Rate.

  • Multicast Address
    This is the multicast address for the data stream. For reference, Ghost uses addresses in the range 224.77.2.0 - 224.77.255.255, which is within the global scope of multicast addresses. The site-local scope of multicast addresses is 239.255.0.0 - 239.255.255.255. Consult your network administrator for a range appropriate for your network. I cannot emphasize this enough, using an inappropriate multicast address can take down your network, and this will not make your network administrator inclined to work with you or allow you to use multicast asr.

  • Client Data Rate
    This is the rate the slowest client can write data to its target in bytes per second. If asr misses data on the first pass and slowing the Data Rate doesn't resolve it, setting the Client Data Rate will dynamically regulate the speed of the multicast stream to allow clients more time to write the data.

  • Loop Suspend
    This imposes a limit to the number of times that the server multicasts the image file after no more clients have initiated a restore. Once exceeded, the server will stop the stream and wait for new clients before multicasting the image file. It defaults to 0 (e.g. never stop multicasting once a client starts the stream), and should not be set to <2.

* A strategy to determine optimal data rates

There are several factors that affect the performance of multicast asr. Among those are:

  • Network bandwidth (including any throttling imposed by network gear)
  • Concurrent network traffic
  • Interference from non-multicast compliant devices or slow devices (e.g. 10Mb printers or PCs)
  • The speed and health of the client machines' hard drives
  • The speed of the processor of your client machine(s)
  • Whether the image is compressed
  • The degree to which varying portions of the image are compressed (some blocks compress better than others, depending on the data on those blocks)

You should also ask your network administrator "how fast is too fast", especially given the time of day (or on which network segment) that imaging is performed. Multicast traffic can easily consume all bandwidth on your network, therefore you are encouraged to work with your network administrator to determine the right multicast settings and data rates for your network.

Rules of thumb: moderate G4 based servers (800+Mhz, Gigabit ethernet) should be able to stream up to 16-20MB/s (although it would take very fast clients to keep up with that). Higher end client computers can probably keep up with a throttled 10MB/s stream, with G4 portables and other consumer level computers able to work in the 4-6MB/s range. Most G3 based clients are probably going to be in the 2-5MB/s range.

The following steps assume that "the sky is the limit." If the sky is not, in fact, the limit, use the limit imposed by your network admin. Otherwise, follow these steps to determine the maximum data rates that work with your network, clients, and disk image. The general approach will be:

  1. Determine the maximum Data Rate your server/client pair can support by multicasting the disk image as a file (the -file argument)
  2. Determine the throttle for your particular disk image using the Client Data Rate once the Data Rate is determined

* Determine your maximum Data Rate

Follow the steps earlier in this article for setting up a Multicast ASR server. Then:

  1. Configure the multicast server with a high Data Rate, such as 18MB/s:

    defaults write /Users/admin/Desktop/asr "Data Rate" -int 18000000

  2. Start the server with your master image:

    sudo asr -source /path/to/master.dmg -server /Users/admin/Desktop/asr.plist

  3. At your client, use asr to restore the multicasted image to a file (vs. to a device):

    sudo asr -source asr://<yourserver> -file /Volumes/<tgtdisk>/master.dmg -erase

    Make sure you restore to the same drive you plan on doing your production restore onto (i.e. don't do your tests with an attached firewire hard drive).

  4. Stop the server and reduce the Data Rate in increments of 500000 (0.5MB/s) until you can file-restore your image without packet loss. Once you have a Data Rate that you can restore a few times in a row, you might back down a little bit more to give yourself some headroom.

* Determine your maximum Client Data Rate

The Client Data Rate basically indicates the maximum data rate at which client machines can receive data and write data to their drives. This is almost entirely dependent on the processor speed of the client machine. As the server doles out portions of the image, it examines the image for particularly compressed portions (that would travel across the network just as quickly, but would take longer for the client to decompress, and thus process), then dynamically reduces its Data Rate to accommodate the client processing time.

  1. To determine the Client Data Rate, you will now restore the disk image to the device at the client (vs. to a file as in the previous section):

    sudo asr -source asr://<yourserver> -target /Volumes/<tgtdisk> -erase

    At some point in the image, the client will probably not be able to keep up. This is due to different parts of the disk image being more compressed than others, and the time it takes the client to expand/copy that compressed section vs. fixed size data. We'll use the Client Data Rate to smooth out these peaks.

  2. Start with a high Client Data Rate (80MB/s for SATA drives, 50MB/s for PATA drives and portables):

    defaults write /tmp/streamconfig "Client Data Rate" -int 80000000

  3. Let the entire image restore. Similar to before, slow the Client Data Rate in increments of 5000000 (5MB/s) until you have a setting that allows your given client to complete a restore in one pass.

Note that the Client Data Rate calculation is based on the disk image itself, so that will need to be repeated for each unique disk image (although once you have a given disk image, slight changes probably won't affect that setting too much, especially if you give yourself some margin in the settings).

* In summary

  • If you have a network capacity cap, start with that
  • Start with a high Data Rate, multicast restore to a file, and slow down until you succeed
  • Set a high Client Data Rate, multicast restore to a disk, and slow down until you succeed
  • Its more beneficial to reduce your Data Rate to insure that all clients need to make only one pass than to go faster and require machines to make more than one pass (especially with larger images)
  • Clients can retrieve up to 32MB of missed data out-of-band, so you have a little flexibility for outlier clients

Article posted on 3/8/2006. Special thanks to "sphns" for contributing the methodology for determining Data Rate and Client Data Rate.

footer shadow