The New Darkwing: a Briefing

Systems staff take the first steps in a major reconfiguration of our vital campus server system: where we're headed and why...

Joe St Sauver, Ph.D.
Director, User Services and Network Applications
joe@uoregon.edu

Susan Hilton
Director, Administrative Services and Computing Facilities
hilton@uoregon.edu

This article is meant to serve as a briefing for members of the campus community who may be interested in learning more about the Computing Center's work in upgrading Darkwing. Let's begin by talking about "classic Darkwing."

Classic Darkwing

Darkwing is a server that's familiar to many UO faculty and staff, although if you're like many users, you may know it only as an email server. In reality, while Darkwing does deliver email for over 15,000 faculty, staff and graduate student accounts, it does quite a bit more, too:

• Darkwing accounts serve as the basis for faculty, staff, and graduate student authentication for a variety of campus resources, including such services as wireless access, dialin access, virtual private network (VPN) access, and the Blackboard teaching and learning system
• Darkwing hosts several hundred virtual web servers, including key web servers such as www.uoregon.edu, the university's primary website, and many others (for a list of all the virtual hosts served from Darkwing, see http://virtual-www.uoregon.edu/ )
• Darkwing delivers streaming audio, via Real and Darwin Streaming Server, for KWAX and other high profile campus content providers
• Darkwing serves over 1300 majordomo-based email mailing lists
• Darkwing offers shell access for users who want to use pine, run statistical packages, compile and run their own programs, or otherwise work at the UNIX percent-sign (%) shell prompt
• Darkwing provides the boot environment and font server needed for X terminals such as those deployed in the basement corridors of McKenzie Hall and elsewhere on campus
• Darkwing acts as a print server for multiple campus print queues
• Darkwing also handles network backups of user files

That's a lot for a single six-year-old server to do. Not surprisingly, most noticeably beginning in 2004, Darkwing began to deliver unsatisfactory performance--in other words, "it got slow."

Monolithic Architecture

Moreover, having all those services on a single monolithic system also meant that if Darkwing did have problems, or if it were simply taken down for scheduled maintenance, the down time would affect many different activities, including mission critical production services such as faculty email or institutional web page delivery.

Running a single monolithic system also has implications for system replacement: if you were to replace a large monolithic system with an even larger monolithic system, you'd be investing in comparatively uncommon hardware with a premium price tag.

For example, Sun's top-of-the-line flagship E25K server starts at just over a million dollars (list price)[1] and it isn't very hard to configure an E25K in a way that would make it cost in excess of three million dollars, not including any significant disk storage. Even a more modest midrange Sun Fire E6900 with just 16 processors and 64GB of RAM has a list price of nearly $800,000 (again not including material disk space). In tight budgetary times, or at any time, that's a lot of money.

A final factor leading us away from monolithic systems is that we know from the commercial sector that it is easy to get to the point where no matter how much money you have, you can't buy a single system that's big enough to handle the aggregate load you may confront. For example, Google (at least as of March 2003) used over 15,000 commodity-class PCs [2] to meet the world's aggregate search engine requirements.

While we're nowhere near the size of Google, it's easy to see the writing on the wall: single monolithic servers don't scale well.

If We Don't Buy a Single Big Box, How Should We Split Up The Load Onto Multiple Smaller Boxes?

We thought about two approaches we could use to divide our load across multiple smaller servers:

Model #1. We could divide things up by user. Taking that approach would involve:

• having each node running all services
• having some way to hash or "divvy up" users onto multiple nodes in a user-transparent way
• being willing to accept the fact that we might need to pay a substantial amount for any licensed product(s), if we needed to purchase and deploy those licensed products on each node
• recognizing that if we needed to add capacity, we would have to shuffle users around
• accepting the fact that it will be hard to evenly share load among nodes.

Model #2. Alternatively, we could divide the load by splitting things up according to service. Doing that sort of model means that:

• we can tailor nodes to the needs of particular services
• we can point traffic for a given service at a particular node via DNS
• we can limit the extent of licensed products we buy
• we can add capacity trans-parently by adding additional back-end nodes behind a load balancer.

Model # 2 is the model the UO has decided to pursue:

Server Building Blocks; Load Balancing

An inherent part of the new model is the use of commodity dual processor AMD Opteron-based 1U rackmount systems. Those dual processor Opteron boxes are the same sort of systems we've been using for a while for the current Academic Compute Intensive Cluster (e.g., the "acad-cl" nodes, see http://acad-cl0.uoregon.edu/ ), where they've proven themselves by delivering tremendous performance at a very reasonable price.[5] We're also looking at adding a less powerful (and less expensive) building block host to our architecture for situations where a dual Opteron, or even a single Opteron, would simply be overkill and unnecessary.

Two, three, or more building block servers, whether Opteron-based or something less powerful, would then sit behind a pair of load balancers (which are themselves simply building block servers, this time configured to run the LVS load balancing software).

Load balancers look at incoming connections along with backend server load, and then route incoming connections to a suitable server, either working round-robin style or by monitoring the load on each backend server and then sending connections to the currently most lightly loaded host.

Use of multiple load-balanced hosts in this fashion gives the university operational resilience by allowing one building block system to be taken out of service for maintenance or upgrades without users even noticing the change; the down side of this approach is that it increases server count (redundant servers are required) and it requires a way to share the load across the servers (e.g., it requires the use of load balancers).

Besides fixing the load problem and giving us critical operational resilience and a clean path for future growth, the new model also eliminates our reliance on expensive proprietary hardware and proprietary operating systems, while also allowing us to avoid the growing cost of maintaining aging hardware and setting the stage for enabling new services such as larger disk quotas.

Building block servers: a DVD drive (left) and two hard drives. Any number of these server units may be mounted on racks and deployed as needed, providing a more flexible model for network service.

Speaking of Disk Quotas, Storage Is Central

Even though Darkwing load manifested itself as general slowness, that slowness was really a function of limitations in the system's I/O (input/output) to disk storage as much as anything.

When you think about disk storage, you probably think about working within a comparatively limited disk quota, such as 100MB on classic Darkwing.

Emerging trends, such as large desktop drives (200-250 gigabyte desktop hard drives are routinely available now), and things such as Google's Gmail service (offering users 1000MB worth of free disk storage for email), meant that our traditional 100MB email quotas on Darkwing were really pretty antiquated. Meeting user expectations requires that we offer disk quotas that are basically an order of magnitude higher than current levels.

Simply buying more disk wouldn't be sufficient, however. At the same time you add disk space, you also need to increase the ability to do disk I/O operations (read operations, write operations, seek operations, etc.), and you also need to increase disk I/O throughput (the ability to shovel data over the wire from a server such as Darkwing to a network file server connected over the network).

It would do no good to increase the amount of disk space users would have if there wasn't increased capacity to read and write data to that disk space, and network capacity to move that data to/from the file server.

Our recent survey of Darkwing users[3], also identified the fact that users would like to be able to more easily restore accidentally damaged or unintentionally deleted files from backups, and the fact that users would also like to be able to get at their Darkwing files via a Microsoft Windows file system (e.g., "CIFS" or "Samba") rather than just via Darkwing's traditional UNIX-oriented NFS.

Still another factor was the desire to be well positioned to move to NFSv4 as soon as possible to minimize potential NFS-related security issues.[4]

Building Upon What We've Already Been Doing

Over the last few years, the Computing Center has been actively experimenting with a variety of new technologies--technologies which have ended up shaping much of what we'll be doing in upgrading Darkwing.

We've already described the Opteron-based building blocks we'll be using; they are one example of a technology that's proven itself and moved from the lab into production.

Similarly, at least for now, Darkwing's disk storage is running on commodity serial ATA[6] disk network attached storage (NAS), storage which looks much like the sort of disk storage proven in the storage of terabytes of data for the ANTC Route Views project.[7]

A third example of moving proven technologies from one project to another can be seen in the use of RedHat Linux-based load balancers[8] in front of multiple systems, as pioneered on campus by the Administrative Services group and the Systems Group in conjunction with deploying Internet Native Banner.

Transition Architecture

The Darkwing upgrade process has already begun, albeit on more of an expedited basis than we might have liked.

The first stage of that upgrade was the migration of user files from Darkwing's old, slow, and bottlenecked disks to new interim network attached storage (NAS) filers.[9]

The interim filers we're currently using do not offer the scalability and feature set we ultimately want, but its deployment is giving us the breathing room we need to complete procurement of the mirrored production filer that will be the core of the new Darkwing architecture. As we write this, the migration of Darkwing user files to the interim filer has been completed.

The next stage of the upgrade involved migration of POP and IMAP (email reading) services from classic Darkwing to a new load-balanced cluster of Opterons.

Moving user POP and IMAP activity to the new cluster requires contacting Darkwing POP and IMAP users, asking them to change their email programs so that instead of POPing or IMAPing against darkwing.uoregon.edu, they access POP as pop.uoregon.edu (or IMAP as imap.uoregon.edu).

At the same time we asked users to make that change, we also asked them to change their outgoing email server name from darkwing.uoregon.edu to smtp.uoregon.edu

At this time, the migration to the new pop.uoregon.edu, imap.uoregon.edu and smtp.uoregon.edu servers has also taken place.

We've also moved IMHO (the "green" web email client) off of classic Darkwing and onto its own server.Because users currently access that service by clicking on "Web Email" from the UO home page, or by visiting http://email.uoregon.edu/ directly, we've been able to accomplish the migration of that service in a way that's transparent to users (except for the fact that green web email is noticeably faster).

For an overview of Darkwing's current stage of development, see the Evolving Darkwing diagram.

The Production (Mirrored) Filers

Concurrently with the steps described above, we're also working on procuring a pair of production network attached storage (NAS) filers that will form the core of the new Darkwing. Because of the size of those filers (8TB, or 8000GB initially, scalable to just under 100TB or a tenth of a petabyte), and because the contents of those filers will be key to university operations, the contents of the filers will be "mirrored," or constantly synchronized, with one server located in the Computing Center and a second server located elsewhere on campus, connected over a private network consisting of multiple channel-bonded gigabit ethernet links supporting jumbo frames.[10]

That mirroring, along with traditional off-site tape backups, will ensure that even if the primary filer were to be destroyed by a catastrophic fire or other accident, the backup filer would still be available to provide access to your files.

Once deployed, the production filer will also offer users easier user-controlled restoration of damaged or accidentally deleted files, and eventually the ability to access files via CIFS/Samba. Once the production filer's in place, disk quotas should increase from 100MB to 250MB, and then, depending on utilization and other factors, upwards from there.

Deployment of the production filers will also be the key event that will allow us to finally decommission oregon.uoregon.edu, our legacy OpenVMS system, and with it, "black" web email ("green" web email will remain available, and we're also looking at a third generation web email client that will support a spell checker and address book, two features noticeably absent from the current green web email).

password.uoregon.edu

We've also recognized that our web-based password changing system, http://password.uoregon.edu/ needed an upgrade.

For those who may not be familiar with password.uoregon.edu, that host provides the web-based interface for making password changes on Darkwing, Gladstone, and Oregon, while also allowing users to do things such as:

• check their disk usage via the web ( http://password.uoregon.edu/quota/ )
• set or disable mail forwarding ( http://password.uoregon.edu/forward/ )
• set their spam filtering preferences ( http://password.uoregon.edu/allowspam/)
• set their viral husk disposition preferences ( http://password.uoregon.edu/husks/ )

Because most users rarely if ever access their Darkwing accounts via ssh, password.uoregon.edu provides an easy web-based interface for accomplishing tasks that would otherwise require ssh and arcane UNIX commands to accomplish.

The upgrade to password.uoregon.edu has been completed at this point, and that system now consists of a redundant load-balanced pair of systems that has far greater capacity than its predecessor.

Moving to a pair of systems for password.uoregon.edu means that down time for that host should now be a thing of the past, and we now have surge capacity to handle situations when many passwords need to be changed in a short time interval.

LDAP Authentication

Still another change that's underway is LDAP-based[11] authentication. Remember that one of the things that Darkwing currently does is to serve as an authentication system for faculty, staff, and graduate students so that the university can control access to resources such as the wireless network, dialin modems, the virtual private network (VPN), and the Blackboard teaching and learning system. Gladstone serves a similar authentication function for undergraduates.

When users supply their Darkwing username and password to access the UO's wireless network, or modems, or Blackboard, we have some confidence that two things are true:

1. They are who they claim to be and are somehow affiliated with the UO, and secondarily
2. They are probably UO faculty, UO staff, or a UO graduate student (although based simply on their ability to login to Darkwing, we don't know which of these they may be, and there's a chance that they might be an undergraduate, e.g., someone using a Darkwing student account specially created for a class)

Note that the second assertion is pretty weak. For example, if we had some service that could be provided only to staff members but not to faculty or graduate students, based on current Darkwing credentials it would be impossible for us to restrict that service to staff only.

Thinking about that, you start to understand the fact that authentication is only one leg of what's often called a three legged "AAA" stool. The three legs of that stool are:

• Authentication ("I know who you are," often accomplished by using a username and secret password),
• Authorization ("I know what you are/what you should be able to do in that role," e.g., I'm a faculty member, or an undergraduate student), and
• Access control ("I have the ability to control what you can access based on what you're authorized to do/see,"--for example, hypothetically one might allow visitors to campus to access some databases but not others, or one might limit access to shared departmental printers to members of that department)

When the LDAP authentication project has been completed, Darkwing users will login just as they currently do, but behind the scenes the authentication process will be handled by the new LDAP server instead of directly by Darkwing.

Darkwing and Gladstone Consolidation

Another thing we want to mention is the eventual merger of Darkwing and Gladstone into a single unified system, with UO users--faculty, staff, graduate and undergraduate students alike--all using email addresses of the format username@uoregon.edu

While that project is still in the early planning stages, the process is greatly facilitated by the fact that we've always had a unified name space on Darkwing, Gladstone, and Oregon (the legacy OpenVMS cluster), partially in anticipation of the day when those systems might be consolidated. This means that if there are two accounts belonging to the same person, such as jdoe@darkwing.uoregon.edu and jdoe@gladstone.uoregon.edu, we don't need to figure out which of two account owners should be allowed to keep "their" username when merging Darkwing and Gladstone accounts.

The biggest question we're currently wrestling with when thinking about merging Darkwing and Gladstone is how to handle the possibility of two filesystems per user, with one tree of files from a user's Darkwing account and another tree of files from that same user's Gladstone account.

Our current thinking is that:

• users who have only a Darkwing account will see no difference; in a converged environment, their default directory will be their normal Darkwing default directory
• users who have only a Gladstone account will see no difference; in a converged environment, their default directory will be their normal Gladstone default directory
• users who have both a Darkwing and a Gladstone account will have their Darkwing directory as their default directory, and their Gladstone directory will live in a subdirectory (perhaps named "gladstone-files" or something similar)

As we work out these and other issues, we'll keep you posted. (Please note that it will be quite some time before we get to that merger project given all the other tasks which need to be handled first.)

Conclusion

We hope this information will help you understand a little about what's currently going on with respect to upgrading Darkwing. Although we've completed a lot of the work, we still have a lot more to do. We know that you may have questions, so feel free to contact either of us (joe@uoregon.edu or hilton@uoregon.edu) with any questions or concerns you may have.

Notes:[back to top]

[1] Sun Fire E25K Server: http://www.sun.com/servers/highend/sunfire_e25k/index.xml

[2] Web Search for a Planet: The Google Cluster Architecture: http://www.computer.org/micro/mi2003/m2022.pdf

[3] Darkwing Faculty/Staff Survey Results: How You Voted: http://cc.uoregon.edu/cnews/spring2004/dwsurvey.html

[4] NFSv4 and NFS Security: http://cc.uoregon.edu/cnews/summer2004/nfsv4.htm

[5] AMD Opteron Processor: http://www.amd.com/us-en/Processors/ProductInformation/0,,30_118_8825,00.html

[6] Serial ATA: http://www.serialata.org/

[7] Route Views Project: http://www.routeviews.org/

[8] IP Load Balancing: http://www.redhat.com/software/rha/cluster/piranha/

[9] See O'Reilly's "Using SANs and NAS": http://www.oreilly.com/catalog/sansnas/

[10] Practical Issues Associated with 9K MTUs: http://darkwing.uoregon.edu/~joe/jumbos/jumbo-frames.ppt (or .pdf )

[11] An excellent LDAP reference is O'Reilly's "LDAP System Administration": http://www.oreilly.com/catalog/ldapsa/index.html