Expect Saves Time on Tedious Command-Line Tasks

by Cameron Laird

Expect can save you time–hours, or even weeks, of time. If you work at the command line, and ever find yourself annoyed at how tedious some task is, you’re a potential Expect customer. Expect automates interactive applications including telnet, ftp, password updates and more.

The cost to start is small, too: modern Mac OS and many popular Linux distributions include it, and convenient installable packages are available at no charge for Windows and all leading Unix releases.

Great for solving problems you shouldn’t have
What qualifies as a “tedious command-line” task, and how much time can you save? If you’re a system or network administrator, plenty of examples will probably come to mind quickly:

1. Resetting all passwords on a particular host, or for a particular account on several hosts, or for multiple accounts on multiple hosts;

2. Logging in to each of fifty different hosts to confirm their patchlevel or that a particular product is installed or how much memory is active in each; or

3. Updating the configuration in one particular mission-critical proprietary product that only has a clumsy command-line interface.

All these chores are common, well-understood, and easy enough to do in isolation. By the dozenth time you’ve entered nearly-identical information, though, you’ll discover how error-prone and time-consuming your own typing is. A smart administrator knows to “script” his work–but many, many applications, especially those which demand a password or related authentication, resist scripting of the usual /bin/sh or myscript.bat sort.

This is where Expect comes to the rescue. If you can get something to work “by hand”, Expect can almost certainly achieve the same, except programmatically. The Expect home site even offers a free record-and-playback utility, autoexpect, that we use to show quick results for our customers.

Another common task packaged as a brief script and freely available on the home page is passmass. Passmass solves the first problem listed above: it updates passwords for accounts on a list of remote hosts.

Expect confusions and corrections
That’s the Expect story: Expect gives you programmatic control over a command-line problem that would otherwise be hard to script. Everything else is just detail.

Expect is so unusual, though, that many of its details are misunderstood. Here are a few of the facts you deserve to know in judging where Expect can be useful to you:

Expect is powerful
Many longtime users of Expect sincerely believe it’s just a password-handling utility, or only available for Linux. In practical applications, they combine it with Perl or shell scripts to get real work done. While Expect “plays nicely” with other languages, they’re unnecessary. Expect itself is actually a portable, powerful general-purpose language, with rich capabilities to do graphical user interfaces, networking, and sophisticated algorithms. Password management just happens to be a strength that almost no other tool offers.

Expect is hard to explain
Not only is Expect different from other things computer people understand–it’s not just a language, it’s more than a tool, and so on–but many of the best uses of Expect are obscure. Introductions to Expect typically focus on a password-updating problem, simply because that’s something almost all readers understand. While almost all administrators have a need for Expect, communicating what it does in the abstract is a challenge. My own first experience with Expect involved automation of configuration of a petrochemical plant; we estimated that Expect saved about fifty hours when compared to our original plan of entering all the data “by hand.” This was a superb return on the investment of a few hours it took to download and learn Expect for the first time. However, there were, at the time, at most a couple hundred sites that could possibly have cared about scripting that particular database from that particular vendor.

The general need to automate a troublesome interface is one almost all of us have, though, when we look at the specifics of our daily work. Expect can help with: testing bar-code interfaces; configuration of network switches; forensic retrieval of data from cellular handsets; auditing operations in an air-traffic control center; and much, much more.

With the wrong introduction, you might think that a “hammer” is only a tool for helping nail roofing shingles, and never realize that it can also put up dry-wall, demolish and remove cracked concrete, tap a delicate spring into place, and so on. Keep Expect’s general power in mind.

Expect isn’t the best tool for every situation
Expect is also like a hammer in that its best uses have changed through time. Changing technology means that some jobs that used to call for a hammer now are often done with a nail gun or power screwdriver.

Much the same is true with Expect. Early in its history–fifteen years ago, say–it was often used to automate e-mail attachments or Web downloads or FTP scripting. Expect still is just as capable in those roles as it ever was; however, in isolation, better tools are now available. wget is more convenient for nearly all the Web scripting Expect once did. wget doesn’t have Expect’s general-purpose capabilities, but for its intended “Web-scraping” applications, it’s hard to beat.

Expect’s original creator, Don Libes of the National Institute of Standards & Technology, has often remarked what a surprise Expect’s longevity has been. He expected other developers would pick up on the Expect idea, and there’d soon be little or no ongoing need for the program itself. Programmers continue to produce annoying and difficult command-line interfaces, though, and Expect continues to be indispensable.

One of Cameron Laird’s specialties for his employer, Phaseit, Inc., is automation and integration of systems that were NOT designed to be automated or integrated.