Armari, Seymour Cray and a giant gold sarcophagus

Seymour Cray and Armari's T3D

Seymour Cray

If you've never heard of Cray, shame on you. Here's a bit on Cray's enigmatic founder for those that haven't attended geek school.

Born in 1926, Seymour Cray started working on computers as an employee of Electronic Research Associates. ERA were a digital computer vendor and created scientific computer systems for research and development. Following his career at ERA he helped form a company called Control Data Corporation. He created several working supercomputer designs at CDC over the years before leaving to startup on his own in 1972, creating Cray Research. Abandoning a design for the CDC 8600 supercomputer that he'd be working on as he left, he started work on a single processor supercomputer that integrated vector processing ability on the CPU, something not done before at that time.

The Cray-1 was a hit and was followed by the Cray-2 and the less successful Cray-3. Both subsequent systems that followed the Cray-1 increased processing power and memory size and bandwidth by orders of magnitude and Seymour was getting into his stride. Parallel vector-able processing (PVP) supercomputers became the company's focus and speciality, their often bespoke installations full of Seymour's flair for their physical design, as well as the silicon and interconnect technology that lay inside. You'll see some of that physical design flair soon.

Personality wise, Seymour was famous for never wanting to offend anyone, but his answers to questions often infuriated and puzzled the questioner. He'd often answer a question just by smiling, before changing tack and controlling the conversation by asking his own questions.

His most famous contributions to computing are his ideas on integrated cooling and integrated vector processing units on custom microprocessors. Realising in 1991 that commodity microprocessors would soon provide as much computing power as his custom designs, he started work on massively parallel processor (MPP) systems using freely available CPUs. He died in 1996 in a car accident, following the year that the Cray T3D shipped, having set in motion the modern supercomputing world, with many of Cray's competitors formed by customers of his systems who used Cray machines to design their own! However, his ideas live on and his vision of MPP systems using commodity CPUs is alive and well in Armari's T3D. If you know of the Sun Starfire server, that's an old Seymour design that he sold off to Sun, back in the day. (Update: that's not quite correct)

Armari's T3D MC512

Armari's T3D is the first ever T3D that rolled off the production line. A series 6 Cray T3D with serial number 1, it's the first of what Cray called the Typhoon, a large chassis that needed a Cray YMP or C90 supercomputer to host the I/O subsystem. The T3D was simply a massive compute engine, feeding the YMP or C90 via a fast interconnect. T3Ds could contain up to 2048 processors, although a 2048 CPU T3D was never built. Serial number 1 is a 512 CPU, liquid cooled behemoth, complete with system boards.

Each system board holds eight DEC Alpha 21064 processors and the connected memory subsystem. The CPUs are EV4 class and run at 150MHz. The EV4 class of Alpha CPUs from DEC has a single 64-bit floating point unit running at core clock speed. The CPU's instruction scheduler could dual-issue a pair of instructions per clock cycle for entry into the pipeline and had 64 general purpose registers (32 integer, 32 floating point).

A massive 1KB of write-back L1 cache was on the CPU and it featured no L2 cache. It could stream data from memory at 300MB/sec, per CPU, and that scaled per chip. Armari's T3D therefore has 150GB/sec of potential CPU-to-memory bandwidth, given all CPUs accessing their local memory at the same time. Inter-CPU bandwidth is the same, at 300MB/sec, so the CPUs could pass data to each other at high speed. CPU-to-CPU latency was less than 2µs. The CPU topology was much like a folded torus. Each CPU was connected to up to six others using three bi-directional busses in 3D space. If any of a CPU's bus connections is at the edge of the torus it wraps around to the bus of the CPU on the opposite edge.

CPU's are connected by a proprietary switching fabric, with installed wiring connecting compute boards. Each wire in the loop has a latency of a single clock cycle. In a 512 CPU machine like Armari's T3D, that's a lot of wiring. You'll soon see.

A PSU powers each compute board of 8 CPUs and its liquid cooling loop. I'm not entirely sure of their ratings, or whether there's a direct comparison to the PSUs of today, but I'd imagine they're fairly hefty supplies.

As late as 2001, T3D and T3E systems were still dominating the Top500.org list of publically known supercomputer installations worldwide. Dozens of T3D and T3E systems would be clustered together on the list as Universities, research labs, governments and similar institutions clambered to purchase them from Cray for their computing centres.

As Cray started decommissioning T3D and T3E installations for their customers, as new systems were installed in place or the systems hit the end of their useful life and had no further customers, Cray would have them stripped down for parts or metals, scrapping the chassis' and power supplies, cooling systems and generally reusing very little by all accounts. Chippewa Falls Museum of Industry and Technology (CFMIT) houses a Cray sub-museum and the first T3D was either bound for there or to be scrapped, but Armari negotiated for the chassis and complete system innards, albeit in a deactivated and inoperational state (it'll never work again, sadly).

So serial number 1 sits in the same part of Armari's building as their cluster test area, unassumedly sitting among piles of Supermicro mainboard delivery boxes with its cooling sub-unit out in the hall off the main area. A little bit of computing history, their T3D MC512-8 Series 6 Number 1 is quite a sight. It's way taller than me and I stand at 6ft 2in.

Dan was kind enough to let me snap a few pictures of the exterior, and also the interior wiring between the system boards, along with the PSU area at the rear. I only wish I'd snapped a picture of the system's power switch. Remember those switches for the power system in Jurassic Park, as they try and reboot the park control computer, that they have to pump up to charge? It's like that. I'd love to see Tagan slap one of those on their next ATX supply, just for the nostalgia factor. Enough of my rambling, onto the pictures.