Automated Image Testing

Question

Let us say I'm using a packaged-based approach of imaging for the Mac, and have a freshly imaged machine (or better, several different types of machines) ready to test. How do I perform automated testing?

edit:

I am primarily concerned with images for client machines. The testing I have in mind is along these lines:

• Run all the applications that are installed (in /Applicaions), one at a time. Does anything crash? Do applications ask for license codes (and is this expected or not)?
• Even better would be to test loading and saving files in different applications.
• Better still would be to do the above, when logged in under a network account. [A number of programs operate funny for a network user.]
• If possible, it would be nice to know how responsive an application is (but that may be beyond the scope of what could be reasonably tested).

Perhaps more fundamentally:

• Does the system even boot up?
• Does it connect to our network?
• It would also be nice to check that expected services (ARD, SSH) are up and running.
• Is the required software installed?
• Is the hard drive functional?

Answer 1

I've seen Eggplant suggested many times for a way to automate testing - especially with a GUI application(s).

Alternatively a short shell script that runs at first launch and removes itself afterwards to ensure that the configuration settings are set up (eg. bound to an Open Directory, can access servers, read certain preferences using PlistBuddy or defaults, check the MCXCache is correct, etc.)

Answer 2

By "test" do you mean "validate that certain packages and programs exist and that they run as expected"?

If so, I can't speak to the Mac-specific parts, but my approach would be to include a little XML-RPC program as part of the image. It doesn't have to be XML-RPC, but that is an approach I've found useful before.

The program would expose an API on the box for various tests. By default it would run when the machine boots, listening on (say) port 8080. Your control script would connect and run a sequence of commands, unit-test style. You get to control the level of exposure, from "does this directory exist" to "what happens when you run photoshop".

Like I say, this is not Mac-savvy, but would be a simple and portable solution.

PS After passing tests, the package could respond to a final command to remove itself from the image :)

Answer 3

AppleScript seems like the obvious choice. Here's a quick start at it (with some notes on the applications that didn't play along in the minimal testing I did) (note: replace reportEvent with something more relevant...):

my TestApp("Address Book")
#my testApp("Aperture") # Aperture doesn't seem to understand activate -- WTF?
#my testApp("Automator") # Automator brings up a sheet when launched -- not sure how to dismiss
my TestApp("Calculator")
my TestApp("Chess")
my TestApp("Dictionary")
# etc...
my reportEvent("OS", "FinishedTest")


on TestApp(appName)
    tell application appName to activate
    delay 1
    tell application "System Events" to count (application processes whose name is appName)
    if result is 0 then
        my reportEvent(appName, "QuitUnexpectedly")
    end if
    tell application appName to quit
    delay 1
    tell application "System Events" to count (application processes whose name is appName)
    if result is not 0 then
        my reportEvent(appName, "FailedToQuit")
    end if
end TestApp

on reportEvent(appName, eventDesc)
    do shell script "curl 'http://server/eventtracker.php?app=" & sanitizeSpaces(appName) & "&event=" & sanitizeSpaces(eventDesc) & "'"
end reportEvent

on sanitizeSpaces(dangerousString)
    set AppleScript's text item delimiters to " "
    set safeString to text items of dangerousString
    set AppleScript's text item delimiters to "_"
    return safeString as Unicode text
end sanitizeSpaces

...save that as an Application out of AppleScript Editor (so it'll run when opened), then add it to a test account's Login Items list (System Preferences -> Accounts pane -> Login Items tab).

I'll defer the problem of getting a network account to autologin at boot. I suspect there's a not-too-hard way to do it (maybe via managed preferences, or just dropping in the relevant settings files by "hand" -- i.e. post-restore script), but I don't have a useful test setup handy at the moment.

Answer 4

I suppose that would depend on what you're trying to test. I would assume that you could use an apple script to automate whatever action you were trying to test.