Category Archives: Processors

Making sense of the Supermicro motherboard attack

There has been a lot of ‘fog of war’ regarding the alleged implantation of Trojan hardware into Supermicro servers at manufacturing time. Other analyses have cast doubt on the story. But do all the pieces pass the sniff test?

In brief, the allegation is that an implant was added at manufacturing time, attached to the Baseboard Management Controller (BMC). When a desktop computer has a problem, common approaches are to reboot it or to reinstall the operating system. However in a datacenter it isn’t possible to physically walk up to the machine to do these things, so the BMC allows administrators to do them over the network.

Crucially, because the BMC has the ability to install the operating system, it can disrupt the process that boots the operating system – and fetch potentially malicious implant code, maybe even over the Internet.

The Bloomberg Businessweek reports are low on technical details, but they do show two interesting things. The first is a picture of the alleged implant. This shows a 6-pin silicon chip inside a roughly 1mm x 2mm ceramic package – as often used for capacitors and other so-called ‘passive’ components, which are typically overlooked.

The other is an animation highlighting this implant chip on a motherboard. Extracting the images from this animation shows the base image is of a Supermicro B1DRi board. As others have noted, this is mounted in a spare footprint between the BMC chip and a Serial-Peripheral Interface (SPI) flash chip that likely contains the BMC’s firmware. Perhaps the animation is an artist’s concept only, but this is just the right place to compromise the BMC.

SPI is a popular format for firmware flash memories – it’s a relatively simple, relatively slow interface, using only four signal wires. Quad SPI (QSPI), a faster version, uses six wires for faster transmission. The Supermicro board here appears to have a QSPI chip, but also a space for an SPI chip as a manufacturing-time option. The alleged implant is mounted in part of the space where the SPI chip would go. Limited interception or modification of SPI communication is something that a medium complexity digital chip (a basic custom chip, or an off-the-shelf programmable CPLD) could do – but not to a great extent. Six pins is enough to intercept the four SPI wires, plus two power. The packaging of this implant would, however, be completely custom.

What can an implant attached to the SPI wires do? The BMC itself is a computer, running an operating system which is stored in the SPI flash chip. The manual for a MBI-6128R-T2 server containing the B1DRi shows it has an AST2400 BMC chip.

The AST2400 uses a relatively old technology – a single-core 400MHz ARM9 CPU, broadly equivalent to a cellphone from the mid 2000s. Its firmware can come via SPI.

I downloaded the B1DRi BMC firmware from the Supermicro website and did some preliminary disassembly. The AST2400 in this firmware appears to run Linux, which is plausible given it supports complicated peripherals such as PCI Express graphics and USB. (It is not news to many of us working in this field that every system already has a Linux operating system running on an ARM CPU, before power is even applied to the main Intel CPUs — but many others may find this surprising).

It is possible that the implant simply replaces the entire BMC firmware, but there is another way.

In order to start its own Linux, AST2400 boots using the U-Boot bootloader. I noticed one of the options is for the AST2400 to pick up its Linux OS over the network (via TFTP or NFS). If (and it’s a substantial if) this is enabled in the AST2400 bootloader, it would not take a huge amount of modification to the SPI contents to divert the boot path so that the BMC fetched its firmware over the network (and potentially the Internet, subject to outbound firewalls).

Once the BMC operating system is compromised, it can then tamper with the main operating system. An obvious path would be to insert malicious code at boot time, via PCI Option ROMs. However, after such vulnerabilities came to light, defenses have been increased in this area.

But there’s another trick a bad BMC can do — it can simply read and write main memory once the machine is booted. The BMC is well-placed to do this, sitting on the PCI Express interconnect since it implements a basic graphics card. This means it potentially has access to large parts of system memory, and so all the data that might be stored on the server. Since the BMC also has access to the network, it’s feasible to exfiltrate that data over the Internet.

So this raises a critical question: how well is the BMC firmware defended? The BMC firmware download contains raw ARM code, and is exactly 32MiB in size. 32MiB is a common size of an SPI flash chip, and suggests this firmware image is written directly to the SPI flash at manufacture without further processing. Additionally, there’s the OpenBMC open source project which supports the AST2400. From what I can find, installing OpenBMC on the AST2400 does not require any code signing or validation process, and so modifying the firmware (for good or ill) looks quite feasible.

Where does this leave us? There are few facts, and much supposition. However, the following scenario does seem to make sense. Let’s assume an implant was added to the motherboard at manufacture time. This needed modification of both the board design, and the robotic component installation process. It intercepts the SPI lines between the flash and the BMC controller. Unless the implant was designed with a very high technology, it may be enough to simply divert the boot process to fetch firmware over the network (either the Internet or a compromised server in the organisation), and all the complex attacks build from there — possibly using PCI Express and/or the BMC for exfiltration.

If the implant is less sophisticated than others have assumed, it may be feasible to block it by firewalling traffic from the BMC — but I can’t see many current owners of such a board wanting to take that risk.

So, finally, what do we learn? In essence, this story seems to pass the sniff test. But it is likely news to many people that their systems are a lot more complex than they thought, and in that complexity can lurk surprising vulnerabilities.

Dr A. Theodore Markettos is a Senior Research Associate in hardware and platform security at the University of Cambridge, Department of Computer Science and Technology.

Efficient multivariate statistical techniques for extracting secrets from electronic devices

That’s the title of my PhD thesis, supervised by Markus Kuhn, which has become available recently as CL tech report 878:

In this thesis I provide a detailed presentation of template attacks, which are considered the most powerful kind of side-channel attacks, and I present several methods for implementing and evaluating this attack efficiently in different scenarios.

These contributions may allow evaluation labs to perform their evaluations faster, show that we can determine almost perfectly an 8-bit target value even when this value is manipulated by a single LOAD instruction (may be the best published results of this kind), and show how to cope with differences across devices, among others.

Some of the datasets used in my experiments along with MATLAB scripts for reproducing my results are available here:


CHERI: Architectural support for the scalable implementation of the principle of least privilege

[CHERI tablet photo]
FPGA-based CHERI prototype tablet — a 64-bit RISC processor that boots CheriBSD, a CHERI-enhanced version of the FreeBSD operating system.
Only slightly overdue, this post is about our recent IEEE Security and Privacy 2015 paper, CHERI: A Hybrid Capability-System Architecture for Scalable Software Compartmentalization. We’ve previously written about how our CHERI processor blends a conventional RISC ISA and processor pipeline design with a capability-system model to provide fine-grained memory protection within virtual address spaces (ISCA 2014, ASPLOS 2015). In our this new paper, we explore how CHERI’s capability-system features can be used to implement fine-grained and scalable application compartmentalisation: many (many) sandboxes within a single UNIX process — a far more efficient and programmer-friendly target for secure software than current architectures.

Continue reading CHERI: Architectural support for the scalable implementation of the principle of least privilege

Four cool new jobs

We’re advertising for four people to join the security group from October.

The first three are for two software engineers to join our new cybercrime centre, to develop new ways of finding bad guys in the terabytes and (soon) petabytes of data we get on spam, phish and other bad stuff online; and a lawyer to explore and define the boundaries of how we share cybercrime data.

The fourth is in Security analysis of semiconductor memory. Could you help us come up with neat new ways of hacking chips? We’ve invented quite a few of these in the past, ranging from optical fault induction through semi-invasive attacks generally. What’s next?

The CHERI capability model: Revisiting RISC in an age of risk (ISCA 2014)

Last week, Jonathan Woodruff presented our joint paper on the CHERI memory model, The CHERI capability model: Revisiting RISC in an age of risk, at the 2014 International Symposium on Computer Architecture (ISCA) in Minneapolis (video, slides). This is our first full paper on Capability Hardware Enhanced RISC Instructions (CHERI), collaborative work between Simon Moore’s and my team composed of members of the Security, Computer Architecture, and Systems Research Groups at the University of Cambridge Computer Laboratory, Peter G. Neumann’s group at the Computer Science Laboratory at SRI International, and Ben Laurie at Google.

CHERI is an instruction-set extension, prototyped via an FPGA-based soft processor core named BERI, that integrates a capability-system model with a conventional memory-management unit (MMU)-based pipeline. Unlike conventional OS-facing MMU-based protection, the CHERI protection and security models are aimed at compilers and applications. CHERI provides efficient, robust, compiler-driven, hardware-supported, and fine-grained memory protection and software compartmentalisation (sandboxing) within, rather than between, addresses spaces. We run a version of FreeBSD that has been adapted to support the hardware capability model (CheriBSD) compiled with a CHERI-aware Clang/LLVM that supports C pointer integrity, bounds checking, and capability-based protection and delegation. CheriBSD also supports a higher-level hardware-software security model permitting sandboxing of application components within an address space based on capabilities and a Call/Return mechanism supporting mutual distrust.

The approach draws inspiration from Capsicum, our OS-facing hybrid capability-system model now shipping in FreeBSD and available as a patch for Linux courtesy Google. We found that capability-system approaches matched extremely well with least-privilege oriented software compartmentalisation, in which programs are broken up into sandboxed components to mitigate the effects of exploited vulnerabilities. CHERI similarly merges research capability-system ideas with a conventional RISC processor design, making accessible the security and robustness benefits of the former, while retaining software compatibility with the latter. In the paper, we contrast our approach with a number of others including Intel’s forthcoming Memory Protection eXtensions (MPX), but in particular pursue a RISC-oriented design instantiated against the 64-bit MIPS ISA, but the ideas should be portable to other RISC ISAs such as ARMv8 and RISC-V.

Our hardware prototype is implemented in Bluespec System Verilog, a high-level hardware description language (HDL) that makes it easier to perform design-space exploration. To facilitate both reproducibility for this work, and also future hardware-software research, we’ve open sourced the underlying Bluespec Extensible RISC Implementation (BERI), our CHERI extensions, and a complete software stack: operating system, compiler, and so on. In fact, support for the underlying 64-bit RISC platform, which implements a version of the 64-bit MIPS ISA, was upstreamed to FreeBSD 10.0, which shipped earlier this year. Our capability-enhanced versions of FreeBSD (CheriBSD) and Clang/LLVM are distributed via GitHub.

You can learn more about CHERI, BERI, and our larger clean-slate hardware-software agenda on the CTSRD Project Website. There, you will find copies of our prior workshop papers, full Bluespec source code for the FPGA processor design, hardware build instructions for our FPGA-based tablet, downloadable CheriBSD images, software source code, and also our recent technical report, Capability Hardware Enhanced RISC Instructions: CHERI Instruction-Set Architecture, and Jon Woodruff’s PhD dissertation on CHERI.

Jonathan Woodruff, Robert N. M. Watson, David Chisnall, Simon W. Moore, Jonathan Anderson, Brooks Davis, Ben Laurie, Peter G. Neumann, Robert Norton, and Michael Roe. The CHERI capability model: Revisiting RISC in an age of risk, Proceedings of the 41st International Symposium on Computer Architecture (ISCA 2014), Minneapolis, MN, USA, June 14–16, 2014.

Research Assistants and Associates in OS, Compiler and CPU Security

We are pleased to announce a job ad for two new research assistants or post-doctoral research associates working on our CTSRD Project, whose target research areas include OS, compiler, and CPU security. This is a joint project between the University of Cambridge’s Security, NetOS, and Computer Architecture research groups, as well as the Computer Science Laboratory at SRI International.

Research Assistants and Associates in OS, Compiler and CPU Security
Fixed-term: The funds for this post are available for 18 months in the first instance.

We are seeking multiple Research Assistants and Post-Doctoral Research Associates to join the CTSRD Project, which is investigating fundamental improvements to CPU-architecture, operating-system (OS), program-analysis, and programming-language structure in support of computer security. The CTSRD Project is a collaboration between the University of Cambridge and SRI International, and part of the DARPA CRASH research programme on clean-slate computer system design for security. More information may be found at:

This position will be an integral part of an international team of researchers spanning multiple institutions in academia and industry. Successful candidates will contribute to the larger research effort by performing system-software, compiler, and hardware implementation and experimentation, developing and evaluating novel hypotheses about refinements to the vertical hardware-software stack. Possible areas of responsibility include: modifying OS kernels (e.g., FreeBSD), adapting compiler suites (e.g., Clang/LLVM); extending an open-source Bluespec-based research-processor design (CHERI); supporting an early-adopter user community for open-source hardware and software; and improving the quality and performance of hardware-software prototypes. The successful candidate must be willing to travel in the UK and abroad engaging with downstream user communities.
Continue reading Research Assistants and Associates in OS, Compiler and CPU Security

Job ad: pre- and post-doctoral posts in processor, operating system, and compiler security

The CTSRD Project is advertising two posts in processor, operating system, and compiler security. The first is a research assistant position, suitable for candidates who may not have a research background, and the second is a post-doctoral research associate position suitable for candidates who have completed (or will shortly complete) a PhD in computer science or a related field.

The CTSRD Project is investigating fundamental improvements to CPU architecture, operating system (OS) design, and programming language structure in support of computer security. The project is a collaboration between the University of Cambridge and SRI International, and part of the DARPA CRASH research programme on clean-slate computer system design.

These positions will be integral parts of an international team of researchers spanning multiple institutions across academia and industry. Successful candidate will provide support for the larger research effort by contributing to low-level hardware and system-software implementation and experimentation. Responsibilities will include extending Bluespec-based CHERI processor designs, modifying operating system kernels and compiler suites, administering test and development systems, as well as performing performance measurements. The position will also support and engage with early adopter communities for our open-source research platform in the UK and abroad.

Candidates should have strong experience with at least one of Bluespec HDL, OS kernel development (FreeBSD preferred), or compiler internals (LLVM preferred); strong experience with the C programming language and use of revision control in large, collaborative projects is essential. Some experience with computer security and formal methods is also recommended.

Further details on the two posts may be found in job ads NR27772 and NR27782. E-mail queries may be sent directly to Dr Robert N. M. Watson.

Both posts are intended to start on 8 July 2013; applications must be received by 9 May 2013.

Interviews on the clean-slate design argument

Over the past two years, Peter G. Neumann and I, along with a host of collaborators at SRI International and the University of Cambridge Computer Laboratory, have been pursuing CTSRD, a joint computer-security research project exploring fundamental revisions to CPU design, operating systems, and application program structure. Recently we’ve been talking about the social, economic, and technical context for that work in a series of media interviews, including one with ACM Queue on research into the hardware-software interface posted previously.

A key aspect to our argument is that the computer industry has been pursuing a strategy of hill climbing with respect to security; if we were willing to take a step back and revisit some of our more fundamental design choices, learning from longer-term security research over the last forty years, then we might be able to break aspects of the asymmetry driving the current arms race between attackers and defenders. This clean-slate argument doesn’t mean we need to throw everything away, but does suggest that more radical change is required than is being widely considered, as we explore in two further interviews:

CFP: Runtime Environments, Systems, Layering and Virtualized Environments (RESoLVE 2013)

This year, we presented two papers at RESoLVE 2012 relating to the structure of operating systems and hardware, one focused on CPU instruction set security features out of our CTSRD project, and another on efficient and reconfigurable communications in data centres out of our MRC2 project.

I’m pleased to announce the Call for Papers for RESoLVE 2013, a workshop (co-located with ASPLOS 2013) that brings together researchers in both the OS and language level virtual machine communities to exchange ideas and experiences, and to discuss how these separate layers can take advantage of each others’ services. This has a particular interest to the security community, who both want to build, and build on, security properties spanning hardware protection (e.g., VMs) and language-level protection.

Runtime Environments, Systems, Layering and Virtualized Environments
(RESoLVE 2013)

ASPLOS 2013 Workshop, Houston, Texas, USA
March 16, 2013


Today’s applications typically target high-level runtime systems and frameworks. At the same time, the operating systems on which they run are themselves increasingly being deployed on top of (hardware) virtual machines. These trends are enabling applications to be written, tested, and deployed more quickly, while simplifying tasks such as checkpointing, providing fault-tolerance, enabling data and computation migration, and making better, more power-efficient use of hardware infrastructure.

However, much current work on virtualization still focuses on running unmodified legacy systems and most higher-level runtime systems ignore the fact that they are deployed in virtual environments. The workshop on Runtime Environments, Systems, Layering, and Virtualized Environments (RESoLVE 2013) aims to brings together researchers in both the OS and language level virtual machine communities to exchange ideas and experiences and to discuss how these separate layers can take advantage of each others’ services.

Continue reading CFP: Runtime Environments, Systems, Layering and Virtualized Environments (RESoLVE 2013)

ACM Queue interview on research into the hardware-software interface

ACM Queue has posted my August 2012 interview on research into the hardware-software interface. We discuss the importance of a whole-stack view in addressing contemporary application security problems, which are often grounded in how we represent and execute software over lower-level substrates. We need to consider CPU design, operating systems, programming languages, applications, and formal methods — which requires building collaborations that span traditional silos in computer science research. I also consider the impact of open source on software security research methodology, and how we might extend those ideas to CPU research. A motivation for this investigation is our experimental CHERI hybrid capability processor, part of the CTSRD Project, a long-term research collaboration between the security, operating systems, and computer architecture groups at the University of Cambridge Computer Laboratory and the systems and formal methods groups SRI International Computer Science Laboratory.