= Running Barrelfish on Gem5 =

There is support for booting Barrelfish on the [[http://www.gem5.org|gem5 simulator]].

This provides a platform for experimenting with OS/architecture co-design, and for experimenting with architectural support for aspects of Barrelfish (e.g., message passing).  It is not intended as a general-purpose platform for experimenting with Barrelfish -- QEMU is typically much faster.

== Running ARMv7 Barrelfish on Gem5 ==

Barrelfish for ARM on Gem5 supports up to 4 cores and caches. There are some issues with the O3 CPU model and multiple cores though. Below are instructions to set up gem5 and boot ARM Barrelfish on it.

1. Build Barrelfish for ARMv7 on Gem5
  * Run `<srcdir>/hake/hake.sh -s <srcdir> -a armv7` in your build directory
  * Run `make VExpressEMM-A15`

2. Get the gem5 simulator according to the instructions on http://gem5.org/Download.  Our nightly builds are currently using revision 629fe6e6c781 from the mercurial mirror (repo.gem5.org/gem5), but generally upstream gem5 should work fine. git commit `8e8ad9d` is known to work (state: 2017-07-24).

3. (optional) If you want to use the test harness, apply the following patch to gem5 before building it:

{{{
diff --git a/configs/common/Options.py b/configs/common/Options.py
index ea7c75b..defdc40 100644
--- a/configs/common/Options.py
+++ b/configs/common/Options.py
@@ -80,6 +80,9 @@ def addNoISAOptions(parser):
     parser.add_option("--mem-type", type="choice", default="DDR3_1600_8x8",
                       choices=MemConfig.mem_names(),
                       help = "type of memory to use")
+    parser.add_option("--console-port", action="store", type="int",
+                      default=3456,
+                      help = """TCP port for the system console.""")
     parser.add_option("--mem-channels", type="int", default=1,
                       help = "number of memory channels")
     parser.add_option("--mem-ranks", type="int", default=None,
diff --git a/configs/example/fs.py b/configs/example/fs.py
index 6507937..7df4aa5 100644
--- a/configs/example/fs.py
+++ b/configs/example/fs.py
@@ -341,6 +341,7 @@ else:
 np = options.num_cpus
 
 test_sys = build_test_system(np)
+test_sys.terminal.port = options.console_port
 if len(bm) == 2:
     drive_sys = build_drive_system(np)
     root = makeDualRoot(True, test_sys, drive_sys, options.etherdump)
}}}

This allows the test harness to choose an arbitrary port for the telnet console to avoid conflicts when multiple builds are running concurrently.


4. Build gem5 with `scons build/ARM/gem5.fast`

5. Set the M5_PATH environment variable to tell the Barrelfish tooling where to find gem5.
 * `export M5_PATH=/path/to/gem5`

6. Run the simulation:
 * `make gem5_armv7_vexpressemm`

7. To get the output of Barrelfish you can use `telnet localhost 3456`. If you see (approximately) the following, you've successfully run the default configuration:
{{{
Telnet escape character is '^E'.
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^E'.
==== m5 slave terminal: Terminal 0 ====
kernel 0: ARMv7-A: Boot driver invoked as: /armv7/sbin/cpu_a15ve loglevel=3 periphbase=0x2c000000 consolePort=0 timerirq=29
kernel 0: ARMv7-A: This is an ARM Cortex-A15 r0p0
kernel 0: ARMv7-A:   Security extensions implemented
kernel 0: ARMv7-A:   Virtualisation extensions implemented.
kernel 0: ARMv7-A:   Generic timer not implemented.
kernel 0: ARMv7-A: First byte of boot driver at 0x80000000
kernel 0: ARMv7-A: First byte of CPU driver at 0x80015000
kernel 0: ARMv7-A: CPU driver entry point is 0x80015000
kernel 0: ARMv7-A: Boot data structure at kernel VA 80008400
kernel 0: ARMv7-A: Initialising kernel page tables.
kernel 0: ARMv7-A: Initialising kernel paging, using RAM at 80000000-9fffffff
kernel 0: ARMv7-A: Switching page tables and jumping - see you in arch_init().
kernel 0: ARMv7-A:  MMU is currently disabled.
kernel 0: ARMv7-A:  Alignment checking is currently disabled.
kernel 0: ARMv7-A:  Caches are currently disabled.
kernel 0: ARMv7-A: Barrelfish CPU driver starting on ARMv7
kernel 0: ARMv7-A: Core data at 0x80008400
kernel 0: ARMv7-A: Global data at 0x80008524
kernel 0: ARMv7-A: Boot record at 0x0
kernel 0: ARMv7-A: Build ID: ab658a2177061b048562b2d4716ca51c2cbebe05
kernel 0: ARMv7-A: Multiboot info:
kernel 0: ARMv7-A:  info header at 0x81a01000
kernel 0: ARMv7-A:  mods_addr is P:0x81a0108c
kernel 0: ARMv7-A:  mods_count is 0x16
kernel 0: ARMv7-A:  cmdline is at P:0x81a0118c
kernel 0: ARMv7-A:  cmdline reads '/armv7/sbin/cpu_a15ve loglevel=3 periphbase=0x2c000000 consolePort=0 timerirq=29'
kernel 0: ARMv7-A:  mmap_length is 0x2
kernel 0: ARMv7-A:  mmap_addr is P:0x81a0105c
kernel 0: ARMv7-A:  multiboot_flags is 0x76
kernel 0: ARMv7-A: Initializing exceptions.
kernel 0: ARMv7-A: Mapping vectors at P:80016000 to ffff0000 using L2 table at P:80008000
kernel 0: ARMv7-A: KCB at 0x80008528
kernel 0: ARMv7-A: Parsing command line
kernel 0: ARMv7-A: Debug port initialized.
kernel 0: ARMv7-A: Initializing the GIC
kernel 0: GIC: 128 interrupt lines detected
kernel 0: GIC: gic_init done
kernel 0: ARMv7-A: gic_init done
kernel 0: ARMv7-A: 1 cores in system
kernel 0: ARMv7-A: Enabling timers
kernel 0: CortexA15 platform: System counter frequency is 25165824Hz.
kernel 0: CortexA15 platform: Timeslice interrupt every 503316 ticks (0ms).
kernel 0: CortexA15 platform: Timer interrupt is 29
kernel 0: GIC: gic_enable_interrupt for id=0x1d, offset=0x20000000, index=0x0
kernel 0: ARMv7-A: Enabling cycle counter user access
kernel 0: ARMv7-A: Enabling VFP
kernel 0: ARMv7-A: Setting coreboot spawn handler
kernel 0: ARMv7-A: Calling arm_kernel_startup
kernel 0: ARMv7-A: arm_kernel_startup entered
kernel 0: ARMv7-A: Doing BSP related bootup
kernel 0: ARMv7-A: spawn_bsp_init
kernel 0: ARMv7-A: spawn_init_common armv7/sbin/init
spawn module: armv7/sbin/init
kernel 0: ARMv7-A: init_page_tables done: init_l1=0x81ba0000 init_l2=0x81ba4000
kernel 0: ARMv7-A: Calling paging_context_switch with address = 81ba0000
kernel 0: ARMv7-A: about to call mem_to_local_phys with lvaddr=81ba0000
kernel 0: ARMv7-A: Using device region at PA:0x0-0x7fffffff
kernel 0: ARMv7-A: spawn_init_common: starting from=224000
Dump of dispatcher at address 0x81a40000:
  disabled      = 1 (RESUME)
  haswork       = 0
  udisp         = 0x224000
  lmp_delivered = 0
  lmp_seen      = 0
  lpm_hint      = 0x0
  dispatcher_run                = 0x0
  dispatcher_pagefault          = 0x0
  dispatcher_pagefault_disabled = 0x0
  dispatcher_trap               = 0x0
  systime      = 0x0
  wakeup       = 0x0
  name         = init
  fpu_used     = 0
  fpu_trap     = 1
  curr_core_id = 0x0
kernel 0: ARMv7-A: Trying to enable interrupts
kernel 0: ARMv7-A: Done enabling interrupts
kernel 0: ARMv7-A: Calling dispatch from arm_kernel_startup, start address is=204000
init: invoked as: init 2097152
init.0.0: using 0x9f000000, 16 MB for init's allocator
Spawning memory server (armv7/sbin/mem_serv)...
Spawning monitor (armv7/sbin/monitor)...
monitor: invoked as: armv7/sbin/monitor 7217152
RAM allocator initialised, 465 MB (of 465 MB) available
Spawning ramfsd on core 0
ramfsd.0.0: pre-populating from boot image...
ramfsd.0.0: unpacking Gzipped CPIO /eclipseclp_ramfs.cpio.gz
ramfsd.0.0: unpacking Gzipped CPIO /skb_ramfs.cpio.gz
ramfsd.0.0: ready
Spawning skb on core 0
}}}

The output will appear to hang here for ~15min, as initializing the SKB takes a while on gem5.

{{{
monitor.0.0: Storing BSP KCB in octopus
monitor.0.0: octopus_set_bspkcb: storing cap
monitor.0.0: octopus_set_bspkcb: setting mapping
Spawning /armv7/sbin/kaluga on core 0
Spawning /armv7/sbin/spawnd on core 0
Spawning /armv7/sbin/startd on core 0
Kaluga using additional device_db file: plat_VE_A15x1.
spawnd.0 up.
kaluga.0.0: CPU driver reports 1 core(s).
Boot driver not found. Do not boot discovered CPU 0.
startd.0.0: starting app /armv7/sbin/serial_kernel on core 0
kaluga.0.0: Kaluga running on VExpressEMM
spawnd.0.0: spawning /armv7/sbin/serial_kernel on core 0
startd.0.0: starting app /armv7/sbin/angler on core 0
kernel 0: GIC: gic_enable_interrupt for id=0x25, offset=0x20, index=0x1
spawnd.0.0: spawning /armv7/sbin/angler on core 0
startd.0.0: starting app /armv7/sbin/memtest on core 0
kernel 0: type = 44, bytes = 0
kernel 0: destcn=800, dest_level=1, dest_slot=44
spawnd.0.0: spawning /armv7/sbin/memtest on core 0
No bootscript
memtest passed successfully! on core 0
spawnd.0.0: spawning /armv7/sbin/fish on core 0
fish v0.2 -- pleased to meet you!
File /init.fish not found
available commands:
help           print_cspace   quit           nproc          ps
demo           pixels         mnfs           oncore         reset
poweroff       skb            mount          ls             cd
pwd            touch          cat            hd             cat2
dd             cp             rm             mkdir          rmdir
setenv         src            printenv       free           dump_caps
>
}}}

NOTE 1: If you use --cpu-type=arm_detailed, the simulation takes a long time (depending on your machine up to an hour just to boot Barrelfish) 

NOTE 2: The following warnings seem to be benign, although they may point to Barrelfish implementation that may be incomplete:
{{{
warn: Existing EnergyCtrl, but no enabled DVFSHandler found.
warn: The clidr register always reports 0 caches.
warn: clidr LoUIS field of 0b001 to match current ARM implementations.
warn:   instruction 'mcr dccmvau' unimplemented
warn: Ignoring write to miscreg pmuserenr
warn: Ignoring write to miscreg pmintenclr
}}}

=== Old steps ===

If you encounter problems building gem5, you may want to try with the following patch applied (Thanks to Mehdi Amiri for providing it!)

{{{
diff --git a/SConstruct b/SConstruct
--- a/SConstruct
+++ b/SConstruct
@@ -510,7 +510,7 @@
 if main['GCC']:
     main.Append(CCFLAGS=['-pipe'])
     main.Append(CCFLAGS=['-fno-strict-aliasing'])
-    main.Append(CCFLAGS=['-Wall', '-Wno-sign-compare', '-Wundef'])
+    # main.Append(CCFLAGS=['-Wall', '-Wno-sign-compare', '-Wundef'])
     # Read the GCC version to check for versions with bugs
     # Note CCVERSION doesn't work here because it is run with the CC
     # before we override it from the command line
@@ -566,7 +566,7 @@
 
     main.Append(CCFLAGS=['-pipe'])
     main.Append(CCFLAGS=['-fno-strict-aliasing'])
-    main.Append(CCFLAGS=['-Wall', '-Wno-sign-compare', '-Wundef'])
+    #main.Append(CCFLAGS=['-Wall', '-Wno-sign-compare', '-Wundef'])
     main.Append(CCFLAGS=['-Wno-tautological-compare'])
     main.Append(CCFLAGS=['-Wno-self-assign'])
     # Ruby makes frequent use of extraneous parantheses in the printing
diff --git a/src/python/m5/__init__.py b/src/python/m5/__init__.py
--- a/src/python/m5/__init__.py
+++ b/src/python/m5/__init__.py
@@ -32,7 +32,7 @@
 
 try:
     # Try to import something that's generated by swig
-    import internal
+    import internal.core
 
     # Try to grab something from it in case demandimport is being used
     internal.core.curTick
diff --git a/src/python/m5/internal/__init__.py b/src/python/m5/internal/__init__.py
--- a/src/python/m5/internal/__init__.py
+++ b/src/python/m5/internal/__init__.py
@@ -26,9 +26,9 @@
 #
 # Authors: Nathan Binkert
 
-import core
-import debug
-import event
-import stats
-import trace
-from drain import DrainManager, Drainable
+#import core
+#import debug
+#import event
+#import stats
+#import trace
+#from drain import DrainManager, Drainable
}}}

NOTE: To change the system configuration, adjust 'tools/arm_gem5/gem5script.py' accordingly or change the command to start gem5 in 'symbolic_targets.mk'


=== Old Instructions ===

'''Note:''' As of release2013-01-11, the patch described in step 3 is not needed anymore, and hence we are not bound to a specific version of gem5. At the time of this writting the latest known working version was 0fea324c832c.

1. Build Barrelfish for Arm on Gem5 
  * since release2012-10-03: use 'armv7' as a make target
  * for earlier releases: use 'arm_gem5' instead

2. Get the gem5 simulator with 'hg clone http://repo.gem5.org/gem5 -r d45a02bd5391 gem5' (note you have to use this release to get a working environment out of the box)

3. Change to your gem5 directory and apply 'gem5_patches.patch', located in '/path/to/barrelfish/tools/arm_gem5' to the gem5 source code

4. Build gem5 with 'scons build/ARM/gem5.fast' and add the binary to the PATH 
NOTE: I had to use an older version of gcc, gcc 4.4 - 4.6 worked for me. If you have troubles, install gcc 4.4 and compile with 'CXX=g++-4.4 scons build/ARM/gem5.fast'

5. Run the simulation (do not forget to add the directory build/ARM from within the gem5 directory to the PATH):
 * make arm_gem5

6. To get the output of Barrelfish you can use 'telnet localhost 3456'

NOTE 1: To change the system configuration, adjust 'tools/arm_gem5/gem5script.py' accordingly or change the command to start gem5 in 'symbolic_targets.mk'

NOTE 2: If you use --cpu-type=arm_detailed (use make arm_gem5_detailed), the simulation takes a long time (depending on your machine up to an hour just to boot Barrelfish) 


== Running x86-64 Barrelfish on Gem5 (Old!) ==

The only configuration that has been exercised is full-system x86 using the simple CPU model.  Other x86 configurations may work (e.g., in-order CPU, out-of-order CPU, depending on the underlying implementation of these CPU models in gem5).  I have used configurations both with and without Ruby.

To get started with Barrelfish on gem5:

1. Build a gem5 tree with x86 simple CPU.  I will assume this produces a binary m5.fast, and that this is added to the path.

2. Build an x86_64 Barrelfish tree as usual.

3. In the build directory, run "make m5_kernel".  This will construct a single kernel image file that combines the actual kernel along with the modules and a rudimentary boot-loader.  The output file is called "m5_kernel".  The file "menu.lst.m5" configures which modules are included, and what the physical memory layout of the simulated machine will be.  Re-run "make m5_kernel" after changing "menu.lst.m5".

4. In the build directory, run "make m5".  This will invoke m5.fast, passing it the m5_kernel and a simple machine configuration taken from "m5script.py".  

NB: steps 2--4 can be combined by just running "make m5".  I've listed them separately here to help troubleshooting if problems emerge.

In practice the file m5script.py (and m5.fast command line) will need to be tailored to the particular kind of system that you want to simulate.  These examples are intended just to help with getting started.

Feel free to contact [[mailto:tharris@microsoft.com|Tim Harris]] with any questions.