Running Barrelfish on Gem5
There is support for booting Barrelfish on the gem5 simulator.
This provides a platform for experimenting with OS/architecture co-design, and for experimenting with architectural support for aspects of Barrelfish (e.g., message passing). It is not intended as a general-purpose platform for experimenting with Barrelfish -- QEMU is typically much faster.
Running ARMv7 Barrelfish on Gem5
Barrelfish for ARM on Gem5 supports up to 4 cores and caches. There are some issues with the O3 CPU model and multiple cores though. Below are instructions to set up gem5 and boot ARM Barrelfish on it.
1. Build Barrelfish for ARMv7 on Gem5
Run <srcdir>/hake/hake.sh -s <srcdir> -a armv7 in your build directory
Run make VExpressEMM-A15
2. Get the gem5 simulator according to the instructions on http://gem5.org/Download. Our nightly builds are currently using revision 629fe6e6c781 from the mercurial mirror (repo.gem5.org/gem5), but generally upstream gem5 should work fine. git commit 8e8ad9d is known to work (state: 2017-07-24).
3. (optional) If you want to use the test harness, apply the following patch to gem5 before building it:
diff --git a/configs/common/Options.py b/configs/common/Options.py index ea7c75b..defdc40 100644 --- a/configs/common/Options.py +++ b/configs/common/Options.py @@ -80,6 +80,9 @@ def addNoISAOptions(parser): parser.add_option("--mem-type", type="choice", default="DDR3_1600_8x8", choices=MemConfig.mem_names(), help = "type of memory to use") + parser.add_option("--console-port", action="store", type="int", + default=3456, + help = """TCP port for the system console.""") parser.add_option("--mem-channels", type="int", default=1, help = "number of memory channels") parser.add_option("--mem-ranks", type="int", default=None, diff --git a/configs/example/fs.py b/configs/example/fs.py index 6507937..7df4aa5 100644 --- a/configs/example/fs.py +++ b/configs/example/fs.py @@ -341,6 +341,7 @@ else: np = options.num_cpus test_sys = build_test_system(np) +test_sys.terminal.port = options.console_port if len(bm) == 2: drive_sys = build_drive_system(np) root = makeDualRoot(True, test_sys, drive_sys, options.etherdump)
This allows the test harness to choose an arbitrary port for the telnet console to avoid conflicts when multiple builds are running concurrently.
4. Build gem5 with scons build/ARM/gem5.fast
5. Set the M5_PATH environment variable to tell the Barrelfish tooling where to find gem5.
6. Run the simulation:
7. To get the output of Barrelfish you can use telnet localhost 3456. If you see (approximately) the following, you've successfully run the default configuration:
Telnet escape character is '^E'. Trying 127.0.0.1... Connected to localhost. Escape character is '^E'. ==== m5 slave terminal: Terminal 0 ==== kernel 0: ARMv7-A: Boot driver invoked as: /armv7/sbin/cpu_a15ve loglevel=3 periphbase=0x2c000000 consolePort=0 timerirq=29 kernel 0: ARMv7-A: This is an ARM Cortex-A15 r0p0 kernel 0: ARMv7-A: Security extensions implemented kernel 0: ARMv7-A: Virtualisation extensions implemented. kernel 0: ARMv7-A: Generic timer not implemented. kernel 0: ARMv7-A: First byte of boot driver at 0x80000000 kernel 0: ARMv7-A: First byte of CPU driver at 0x80015000 kernel 0: ARMv7-A: CPU driver entry point is 0x80015000 kernel 0: ARMv7-A: Boot data structure at kernel VA 80008400 kernel 0: ARMv7-A: Initialising kernel page tables. kernel 0: ARMv7-A: Initialising kernel paging, using RAM at 80000000-9fffffff kernel 0: ARMv7-A: Switching page tables and jumping - see you in arch_init(). kernel 0: ARMv7-A: MMU is currently disabled. kernel 0: ARMv7-A: Alignment checking is currently disabled. kernel 0: ARMv7-A: Caches are currently disabled. kernel 0: ARMv7-A: Barrelfish CPU driver starting on ARMv7 kernel 0: ARMv7-A: Core data at 0x80008400 kernel 0: ARMv7-A: Global data at 0x80008524 kernel 0: ARMv7-A: Boot record at 0x0 kernel 0: ARMv7-A: Build ID: ab658a2177061b048562b2d4716ca51c2cbebe05 kernel 0: ARMv7-A: Multiboot info: kernel 0: ARMv7-A: info header at 0x81a01000 kernel 0: ARMv7-A: mods_addr is P:0x81a0108c kernel 0: ARMv7-A: mods_count is 0x16 kernel 0: ARMv7-A: cmdline is at P:0x81a0118c kernel 0: ARMv7-A: cmdline reads '/armv7/sbin/cpu_a15ve loglevel=3 periphbase=0x2c000000 consolePort=0 timerirq=29' kernel 0: ARMv7-A: mmap_length is 0x2 kernel 0: ARMv7-A: mmap_addr is P:0x81a0105c kernel 0: ARMv7-A: multiboot_flags is 0x76 kernel 0: ARMv7-A: Initializing exceptions. kernel 0: ARMv7-A: Mapping vectors at P:80016000 to ffff0000 using L2 table at P:80008000 kernel 0: ARMv7-A: KCB at 0x80008528 kernel 0: ARMv7-A: Parsing command line kernel 0: ARMv7-A: Debug port initialized. kernel 0: ARMv7-A: Initializing the GIC kernel 0: GIC: 128 interrupt lines detected kernel 0: GIC: gic_init done kernel 0: ARMv7-A: gic_init done kernel 0: ARMv7-A: 1 cores in system kernel 0: ARMv7-A: Enabling timers kernel 0: CortexA15 platform: System counter frequency is 25165824Hz. kernel 0: CortexA15 platform: Timeslice interrupt every 503316 ticks (0ms). kernel 0: CortexA15 platform: Timer interrupt is 29 kernel 0: GIC: gic_enable_interrupt for id=0x1d, offset=0x20000000, index=0x0 kernel 0: ARMv7-A: Enabling cycle counter user access kernel 0: ARMv7-A: Enabling VFP kernel 0: ARMv7-A: Setting coreboot spawn handler kernel 0: ARMv7-A: Calling arm_kernel_startup kernel 0: ARMv7-A: arm_kernel_startup entered kernel 0: ARMv7-A: Doing BSP related bootup kernel 0: ARMv7-A: spawn_bsp_init kernel 0: ARMv7-A: spawn_init_common armv7/sbin/init spawn module: armv7/sbin/init kernel 0: ARMv7-A: init_page_tables done: init_l1=0x81ba0000 init_l2=0x81ba4000 kernel 0: ARMv7-A: Calling paging_context_switch with address = 81ba0000 kernel 0: ARMv7-A: about to call mem_to_local_phys with lvaddr=81ba0000 kernel 0: ARMv7-A: Using device region at PA:0x0-0x7fffffff kernel 0: ARMv7-A: spawn_init_common: starting from=224000 Dump of dispatcher at address 0x81a40000: disabled = 1 (RESUME) haswork = 0 udisp = 0x224000 lmp_delivered = 0 lmp_seen = 0 lpm_hint = 0x0 dispatcher_run = 0x0 dispatcher_pagefault = 0x0 dispatcher_pagefault_disabled = 0x0 dispatcher_trap = 0x0 systime = 0x0 wakeup = 0x0 name = init fpu_used = 0 fpu_trap = 1 curr_core_id = 0x0 kernel 0: ARMv7-A: Trying to enable interrupts kernel 0: ARMv7-A: Done enabling interrupts kernel 0: ARMv7-A: Calling dispatch from arm_kernel_startup, start address is=204000 init: invoked as: init 2097152 init.0.0: using 0x9f000000, 16 MB for init's allocator Spawning memory server (armv7/sbin/mem_serv)... Spawning monitor (armv7/sbin/monitor)... monitor: invoked as: armv7/sbin/monitor 7217152 RAM allocator initialised, 465 MB (of 465 MB) available Spawning ramfsd on core 0 ramfsd.0.0: pre-populating from boot image... ramfsd.0.0: unpacking Gzipped CPIO /eclipseclp_ramfs.cpio.gz ramfsd.0.0: unpacking Gzipped CPIO /skb_ramfs.cpio.gz ramfsd.0.0: ready Spawning skb on core 0
The output will appear to hang here for ~15min, as initializing the SKB takes a while on gem5.
monitor.0.0: Storing BSP KCB in octopus monitor.0.0: octopus_set_bspkcb: storing cap monitor.0.0: octopus_set_bspkcb: setting mapping Spawning /armv7/sbin/kaluga on core 0 Spawning /armv7/sbin/spawnd on core 0 Spawning /armv7/sbin/startd on core 0 Kaluga using additional device_db file: plat_VE_A15x1. spawnd.0 up. kaluga.0.0: CPU driver reports 1 core(s). Boot driver not found. Do not boot discovered CPU 0. startd.0.0: starting app /armv7/sbin/serial_kernel on core 0 kaluga.0.0: Kaluga running on VExpressEMM spawnd.0.0: spawning /armv7/sbin/serial_kernel on core 0 startd.0.0: starting app /armv7/sbin/angler on core 0 kernel 0: GIC: gic_enable_interrupt for id=0x25, offset=0x20, index=0x1 spawnd.0.0: spawning /armv7/sbin/angler on core 0 startd.0.0: starting app /armv7/sbin/memtest on core 0 kernel 0: type = 44, bytes = 0 kernel 0: destcn=800, dest_level=1, dest_slot=44 spawnd.0.0: spawning /armv7/sbin/memtest on core 0 No bootscript memtest passed successfully! on core 0 spawnd.0.0: spawning /armv7/sbin/fish on core 0 fish v0.2 -- pleased to meet you! File /init.fish not found available commands: help print_cspace quit nproc ps demo pixels mnfs oncore reset poweroff skb mount ls cd pwd touch cat hd cat2 dd cp rm mkdir rmdir setenv src printenv free dump_caps >
NOTE 1: If you use --cpu-type=arm_detailed, the simulation takes a long time (depending on your machine up to an hour just to boot Barrelfish)
NOTE 2: The following warnings seem to be benign, although they may point to Barrelfish implementation that may be incomplete:
warn: Existing EnergyCtrl, but no enabled DVFSHandler found. warn: The clidr register always reports 0 caches. warn: clidr LoUIS field of 0b001 to match current ARM implementations. warn: instruction 'mcr dccmvau' unimplemented warn: Ignoring write to miscreg pmuserenr warn: Ignoring write to miscreg pmintenclr
If you encounter problems building gem5, you may want to try with the following patch applied (Thanks to Mehdi Amiri for providing it!)
diff --git a/SConstruct b/SConstruct --- a/SConstruct +++ b/SConstruct @@ -510,7 +510,7 @@ if main['GCC']: main.Append(CCFLAGS=['-pipe']) main.Append(CCFLAGS=['-fno-strict-aliasing']) - main.Append(CCFLAGS=['-Wall', '-Wno-sign-compare', '-Wundef']) + # main.Append(CCFLAGS=['-Wall', '-Wno-sign-compare', '-Wundef']) # Read the GCC version to check for versions with bugs # Note CCVERSION doesn't work here because it is run with the CC # before we override it from the command line @@ -566,7 +566,7 @@ main.Append(CCFLAGS=['-pipe']) main.Append(CCFLAGS=['-fno-strict-aliasing']) - main.Append(CCFLAGS=['-Wall', '-Wno-sign-compare', '-Wundef']) + #main.Append(CCFLAGS=['-Wall', '-Wno-sign-compare', '-Wundef']) main.Append(CCFLAGS=['-Wno-tautological-compare']) main.Append(CCFLAGS=['-Wno-self-assign']) # Ruby makes frequent use of extraneous parantheses in the printing diff --git a/src/python/m5/__init__.py b/src/python/m5/__init__.py --- a/src/python/m5/__init__.py +++ b/src/python/m5/__init__.py @@ -32,7 +32,7 @@ try: # Try to import something that's generated by swig - import internal + import internal.core # Try to grab something from it in case demandimport is being used internal.core.curTick diff --git a/src/python/m5/internal/__init__.py b/src/python/m5/internal/__init__.py --- a/src/python/m5/internal/__init__.py +++ b/src/python/m5/internal/__init__.py @@ -26,9 +26,9 @@ # # Authors: Nathan Binkert -import core -import debug -import event -import stats -import trace -from drain import DrainManager, Drainable +#import core +#import debug +#import event +#import stats +#import trace +#from drain import DrainManager, Drainable
NOTE: To change the system configuration, adjust 'tools/arm_gem5/gem5script.py' accordingly or change the command to start gem5 in 'symbolic_targets.mk'
Note: As of release2013-01-11, the patch described in step 3 is not needed anymore, and hence we are not bound to a specific version of gem5. At the time of this writting the latest known working version was 0fea324c832c.
1. Build Barrelfish for Arm on Gem5
- since release2012-10-03: use 'armv7' as a make target
- for earlier releases: use 'arm_gem5' instead
2. Get the gem5 simulator with 'hg clone http://repo.gem5.org/gem5 -r d45a02bd5391 gem5' (note you have to use this release to get a working environment out of the box)
3. Change to your gem5 directory and apply 'gem5_patches.patch', located in '/path/to/barrelfish/tools/arm_gem5' to the gem5 source code
4. Build gem5 with 'scons build/ARM/gem5.fast' and add the binary to the PATH NOTE: I had to use an older version of gcc, gcc 4.4 - 4.6 worked for me. If you have troubles, install gcc 4.4 and compile with 'CXX=g++-4.4 scons build/ARM/gem5.fast'
5. Run the simulation (do not forget to add the directory build/ARM from within the gem5 directory to the PATH):
- make arm_gem5
6. To get the output of Barrelfish you can use 'telnet localhost 3456'
NOTE 1: To change the system configuration, adjust 'tools/arm_gem5/gem5script.py' accordingly or change the command to start gem5 in 'symbolic_targets.mk'
NOTE 2: If you use --cpu-type=arm_detailed (use make arm_gem5_detailed), the simulation takes a long time (depending on your machine up to an hour just to boot Barrelfish)
Running x86-64 Barrelfish on Gem5 (Old!)
The only configuration that has been exercised is full-system x86 using the simple CPU model. Other x86 configurations may work (e.g., in-order CPU, out-of-order CPU, depending on the underlying implementation of these CPU models in gem5). I have used configurations both with and without Ruby.
To get started with Barrelfish on gem5:
1. Build a gem5 tree with x86 simple CPU. I will assume this produces a binary m5.fast, and that this is added to the path.
2. Build an x86_64 Barrelfish tree as usual.
3. In the build directory, run "make m5_kernel". This will construct a single kernel image file that combines the actual kernel along with the modules and a rudimentary boot-loader. The output file is called "m5_kernel". The file "menu.lst.m5" configures which modules are included, and what the physical memory layout of the simulated machine will be. Re-run "make m5_kernel" after changing "menu.lst.m5".
4. In the build directory, run "make m5". This will invoke m5.fast, passing it the m5_kernel and a simple machine configuration taken from "m5script.py".
NB: steps 2--4 can be combined by just running "make m5". I've listed them separately here to help troubleshooting if problems emerge.
In practice the file m5script.py (and m5.fast command line) will need to be tailored to the particular kind of system that you want to simulate. These examples are intended just to help with getting started.
Feel free to contact Tim Harris with any questions.