F1 for CKW - University of Florida

Simulation FaultInjection & Software FaultTolerance Ed Carlisle Outline Background Simulation Fault-Injection Methodology

Results Related Research Process-Level Redundancy Radiation Effects Fault Injection Fault Tolerance Architecture Maintaining Transparency Results & Overhead Conclusions 2 of 35 Radiation Effects

Transient faults (or soft errors) Occur when particles strike a device causing the deposit or removal of energy which inverts transistor state Usually observed as a bit-flip In order to study these effects in the lab, some form of fault injection can be used 3 of 35 Hardware Fault-Injection Using radiation beam or electromagnetic interference

Using probes to introduce voltage or current changes Advantage Similar to what a device would experience in harsh environment Closely resembles real-world effects on device Disadvantages Possible to damage device under test Device under test must be modified to perform injection

4 of 35 Software Fault-Injection Compile-time injection Runtime injection Uses a trigger mechanism to inject faults during execution Faults can be targeted at any software-visible components Advantage

Corrupts an applications instructions during compilation Device under test does not need to be modified Disadvantage Possible to disturb processing workload in unintended ways 5 of 35 Simulation Fault-Injection Fault injection can be performed in simulation of system Advantages

Injections are transparent to target system Simulation offers greatest amount of controllability and observability Disadvantages Building simulation for target device is not a trivial task Faults in physical system may not manifest in simulation Py th on 6 of 35 Fault Tolerance

Usually involves some form of redundancy Hardware Fault-Tolerance Memory and caches can be protected with ECC or parity TMR is one of the most common forms of HW FT Example of TMR (Triple Modular Redundancy) shown below 7 of 35 Fault Tolerance Hardware Fault-Tolerance (contd) Hardware devices can also be fabricated using

processes that are less susceptible to radiation effects Process of radiation hardening devices can be prohibitively expensive and time consuming RadHard devices are generations behind their COTS counterparts in terms of performance and power consumption Software Fault-Tolerance Very cost-effective approach compared to hardware FT Does not require any modification to device architecture Leverages high-performance, low-power commercial offthe-shelf (COTS) components 8 of 35 QUESTIONS? 9 of 35

Nicholas J. Wange, Justin Quek, Todd M. Rafacz, Sanjay J. Patel Univeristy of Illinois at Urbana-Champaign International Conference on Dependable Systems and Networks 2004 CHARACTERIZING THE EFFECTS OF TRANSIENT FAULTS ON A HIGH PERFORMANCE PROCESSOR PIPELINE 10 of 35 Overview Detailed Verilog model created for a microprocessor architecture, similar in complexity to the Alpha 21264 or AMD Athlon Created a methodology for performing fault

injection on a detailed latch-level simulation of a complex processor Studied the propagation and/or masking of faults from the micro-architectural level to the architectural level 11 of 35 Verilog Processor Model Features Alpha ISA subset Speculative instruction scheduling Memory dependence prediction

Sophisticated branch prediction Up to 132 instructions can occupy the 12 stage pipeline 12 of 35 Fault-Injection Methodology A time at which to inject fault is first selected Then the bit to corrupt is randomly selected Injected faults are a single bit-flip of a state element

The trial is monitored for up to 10,000 cycles Randomly selected from 250-300 start points At each cycle, architectural state is verified against noninjected golden execution Trials are placed into four categories depending on the outcome Each experiment consists of 25,000-30,000 trials 13 of 35 Trial Outcome Categories Micro-architectural state match Termination

Premature termination of the workload (execution error) Silent data corruption Occurs when every bit of state in the machine is equivalent to a non-fault-injected simulation Trials that result in software-visible register or memory corruption (data error) Gray area Trial that does not result in failure (termination or silent data corruption) or micro-architectural state match 14 of 35

Results 15 of 35 Results This chart shows which types of state (relative to their contribution of overall state) contribute to silent data corruption and terminated results Register file corruption is the leading cause of silent data corruption (data errors) and terminated (execution errors) outcomes

16 of 35 Results Although noise is present in the graph, a correlation between processor utilization and benign fault rate can be seen As the number of valid instructions (those that will commit results) in the pipeline decreases the benign fault rate increases Benign faults do not affect program correctness 17 of 35

Shortfalls Some instructions of the Alpha ISA were not implemented in the processor model 10,000 cycle limit for monitoring is quite low Certain components were ignored for fault injection Certainly not enough time for most benchmarks to complete These include caches and prediction structures Corrupted registers were considered application failures

However, I have observed in my research that the majority of faults targeted at registers do not affect program execution or output In my research I use the Simics cycle-accurate system simulation environment to perform fault injections into the register file of the Freescale P2020 dual-core PowerPC-based processor 18 of 35 Simics Fault-Injection Workflow Create Simics script to load initial checkpoint Calculate cycles required for execution Run Simics script

Select checkpoint for injection and inject fault Log results and exit Simics Monitor console output to determine outcome Create Simics script to load and execute injected checkpoint Run Simics script

19 of 35 Create checkpoints and exit Simics Simics Simulation Fault-Injection Results Simics simulation does not have the same level of detail needed to perform fault injection at the micro-architectural level, but does allow for register file fault-injection The chart below shows results obtained when injecting single-bit faults into each of the general purpose registers, during a matrix multiplication application 20 of 35 QUESTIONS?

21 of 35 Alex Shye, Joseph Blomstedt, Tipp Moseley, Vijay Janapa Reddi, Daniel A. Connors IEEE Transaction on Dependable and Secure Computing April-June 2009 PLR: A SOFTWARE APPROACH TO TRANSIENT FAULT TOLERANCE FOR MULTICORE ARCHITECTURES 22 of 35 Process-Level Redundancy Similar to TMR hardware fault-tolerance scheme

Creates a set of redundant processes for an application and compares each output to ensure correct execution Leverages multiple processing cores by allowing the operating system to schedule redundant processes to available cores Biggest challenge is maintaining determinism Transparency can be achieved by maintaining userexpected process semantics Does not require any modifications to target application, operating system, or device architecture Important for legacy binaries whose source is no longer available 23 of 35 Sphere of Replication Specifies the boundary for fault detection and containment

Data entering the SoR is replicated All execution within the SoR is redundant Any data leaving the SoR is compared to check for faults Any execution outside the SoR is not protected A typical hardware-centric SoR is shown on the left PLRs software-centric SoR is shown on the right 24 of 35 PLR Components Monitor process Figurehead process

Maintains semantics Master process Slave processes Maintains semantics Redundant processes System call emulation Maintains determinism Responsible for fault detection and recovery

25 of 35 Maintaining Process Semantics Example semantics: Each application is assigned a process identifier (PID) which exists throughout execution and returned to the operating system after completion When an application exits, it returns the correct exit code A signal that is sent to a valid PID will have the intended effects (e.g. SIGKILL will kill the process) Figurehead process

Original process becomes figurehead process after redundant processes are created Does not perform any real work 26 of 35 Maintaining Process Semantics Figurehead process (contd) Sleeps and waits for redundant processes to complete Receives application exit value and exits correctly Responsible for forwarding incoming signals to all redundant processes Monitor process

Certain signals are not easily forwarded A SIGKILL signal would kill the figurehead process, but leave behind all redundant processes Monitor process polls the state of figurehead process If figurehead is killed or stopped, monitor process will kill or stop redundant processes 27 of 35 Maintaining Determinism & Transparency System call emulation unit

Responsible for input replication, output comparison, and system call emulation Responsible for ensuring that redundant processes interacting with the system appear as if only the original process is executing System calls that return nondeterministic data (such as the system time) must be emulated to ensure all processes use the same data Master vs. slave processes System calls that modify any system state are only executed by the master process Other system calls are performed once for the master process and replicated for the slave processes 28 of 35

Fault Detection The system call emulation unit is responsible for providing fault detection and recovery A fault causing the application to hang can be detected by a watchdog timer attached to the emulation unit The timer begins when a processes enters the unit If the rest of processes do not enter the unit within a specified amount of time, an execution error is signaled Faults causing control-flow errors can also be detected if all processes do not request the same system call when entering the emulation unit 29 of 35

Fault Recovery If an output mismatch occurs, a majority vote can be used to kill process producing incorrect data A watchdog timeout can occur in two cases Bad process is then replaced by forking correct process If a faulty process calls the emulation unit while other processes are executing, it is killed and replaced by forking a correct process at the next system call If a faulty process hangs while the other processes are waiting in the emulation unit, it is killed and replaced by a

correct process If a process fails, it is simply replaced by duplicating one of the remaining processes 30 of 35 Results PLR eliminates all failed, abort, and incorrect cases Output comparison converts abort and incorrect cases to mismatches PLR detects failed cases, converting them into sighandler cases A small number of failed cases are detected as mismatch with PLR The mismatch is caught before the application can fail

Some floating-point benchmarks actually caused correct outcomes to become mismatches with PLR enabled The specdiff tool included with the benchmarks uses a tolerance when checking output data, whereas PLRs output comparison checks raw data 31 of 35 Overhead Incurred A) 2 processes B) 3 processes C) 2 processes optimized D) 3 processes optimized Contention overhead is mainly caused by sharing memory bandwidth between redundant processes Emulation overhead is caused by synchronization and transferring/comparing data in shared memory 32 of 35

Shortfalls Functionality of system call emulation unit is detailed, however not many implementation details are provided Replicating results would be hard to accomplish without more specific implementation details Faults occurring during PLR code or operating system execution are not protected against Only supports single-threaded applications May not function as intended if using more redundant processes than physical cores available

Timeouts assume all processes are running concurrently 33 of 35 Conclusions Simulation Fault-Injection Allowed for injections to target areas not accessible to software or hardware fault-injection tools Showed that many faults are masked before they are even visible to software Process-Level Redundancy

Software fault-tolerance scheme Similar to triple modular redundancy hardware scheme Transparent to system and target application Does not require any user intervention to apply protection Able to detect all application failures and incorrect output 34 of 35 QUESTIONS? 35 of 35

Recently Viewed Presentations

  • The Nature of the Teaching Profession

    The Nature of the Teaching Profession

    Characteristics of a Profession…body of knowledge, lengthy training and licensure, specific services to clients, codes of practice, autonomous decisions, relatively free from lay control The Nature of the Teaching Profession Chapter 6 Good Teachers throughout History The Greeks, particularly the...
  • COMPRESSOR BEARING SIMILARITY TEST RIG Steven Kaiser Project

    COMPRESSOR BEARING SIMILARITY TEST RIG Steven Kaiser Project

    The objective of this project was to design a dynamic similarity test rig to replicate the operating conditions of the main crank bearing in the Dresser-Rand ESH-1 air compressor in the Mechanical Engineering machine shop including such characteristics as oil...
  • PowerPoint Presentation

    PowerPoint Presentation

    More complex, clinical diagnosis requirement, metaphylaxis, prophylaxis, Aiming to increase availability. Application process. Pharmacovigilance. Wholesaling . Retailing . Prescribing and dispensing . Cascade and. exceptional imports. P O S S I B L E . F U T U R...
  • Grading and Daylighting - Usda

    Grading and Daylighting - Usda

    GRADING AND DAYLIGHTING The Daylighting commands calculate slope daylighting from a polyline footprint to a surface. A 2D or 3D polyline with elevational information is used to represent the footprint. Additional vertices can be added to the polyline for increased...
  • PowerPoint Presentation

    PowerPoint Presentation

    The Creed . of the . Noncommissioned Officer. C. N. O. o one is more professional than I. I am a Noncommissioned Officer, a leader ofsoldiers. As a Noncommissioned Officer, I realize that I am a member of a time...
  • Choice  6 Steps, 6 Actions, 6 Weeks C

    Choice 6 Steps, 6 Actions, 6 Weeks C

    3.1. Ensure NHS Choices and eRS DOS are up to date with local provider services and notify GPs of any change of providers or services. 3.2. Creation of Choice Information Bank - (pre populated template provided linked to NHS Choices...
  • Soybean Aphids in Iowa Past and Present Marlin

    Soybean Aphids in Iowa Past and Present Marlin

    Soybean Aphids in Iowa - Past and Present Marlin E. Rice Extension Entomologist Iowa State University Acknowledgment Extension Field Specialists-Crops Brian Lang, Decorah (northeast Iowa) Todd Vagts, Carroll (west central Iowa) Soybean Aphids in 2000 (year 1, confirmed counties) Soybean...
  • The Sniper - Mrs. Sullivan

    The Sniper - Mrs. Sullivan

    "The Sniper"Reading Notes. Directions:Label a piece of lined paper with the story's title and author's name at the top. Use this piece of paper to take notes and respond to questions before, during, and after the story. Label each section...