hacks.moe

GitHub | Gitlab | Feed | Privacy Policy | Contact & Impressum

Compiling aarch64 Binaries for gem5 SE Mode

Update April 2019: As of this commit, aarch64 Linux binaries should work as-is in SE mode, even if compiled with glibc. Thank you Ciro for letting me know and thanks to the gem5 developers in general :)

The gem5 simulator has a useful system call emulation mode that lets you run programs without a Linux kernel. The system calls issued by your program are handled by gem5.

Unfortunately, using regular GCC with glibc creates programs that gem5 does not fully support.

~/gem5dir $ aarch64-unknown-linux-gnu-gcc -static hello.c -o hello
~/gem5dir $ build/ARM/gem5.opt configs/example/se.py -c hello
gem5 Simulator System.  http://gem5.org
gem5 is copyrighted software; use the --copyright option for details.

gem5 compiled Sep  8 2018 15:18:08
gem5 started Sep  9 2018 10:45:17
gem5 executing on faui00h, pid 12059
command line: build/ARM/gem5.opt configs/example/se.py -c hello

/proj/ciptmp/ru64tiji/BA/Git/gem5/configs/common/CacheConfig.py:48: \
    SyntaxWarning: import * only allowed at module level
def config_cache(options, system):
Global frequency set at 1000000000000 ticks per second
warn: DRAM device capacity (8192 Mbytes) does not match the address \
    range assigned (512 Mbytes)
0: system.remote_gdb: listening for remote gdb on port 7000
**** REAL SIMULATION ****
info: Entering event queue @ 0.  Starting simulation...
panic: Attempted to execute unimplemented instruction 'mrs' \
    (inst 0x4d5380001)
Memory Usage: 639976 KBytes
Program aborted at tick 266500

The main culprit is the following error.

panic: Attempted to execute unimplemented instruction 'mrs'

To get a working program, you can use an embedded C library that does not use the advanced features glibc uses. I've had success with uClibc.

Using uClibc does require you to build your own tool chain including compiler, but it's all automated and explained well at www.uclibc.org/about.html. Essentially, you're going to do the following:

  1. Download buildroot.
  2. Run make menuconfig.
  3. Set (1) Target Architecture to AArch64 (little endian) and (2) Build options > libraries to static only. If you want C++ support or some other features, there are flags to configure that as well. It's probably a good idea to go through all options, there aren't that many.
  4. Save your config, exit the menu and run make.

After compilation, the directory /output/host/bin contains your just built compilers and binutils for aarch64. Let's try them out!

~/gem5dir $ aarch64-buildroot-linux-uclibc-gcc -static hello.c -o hello
~/gem5dir $ build/ARM/gem5.opt configs/example/se.py -c hello
gem5 Simulator System.  http://gem5.org
gem5 is copyrighted software; use the --copyright option for details.

gem5 compiled Sep  8 2018 15:18:08
gem5 started Sep  9 2018 10:58:26
gem5 executing on faui00h, pid 13513
command line: build/ARM/gem5.opt configs/example/se.py -c hello

/proj/ciptmp/ru64tiji/BA/Git/gem5/configs/common/CacheConfig.py:48: \
    SyntaxWarning: import * only allowed at module level
def config_cache(options, system):
Global frequency set at 1000000000000 ticks per second
warn: DRAM device capacity (8192 Mbytes) does not match the address \
    range assigned (512 Mbytes)
warn: Unknown operating system; assuming Linux.
0: system.remote_gdb: listening for remote gdb on port 7000
**** REAL SIMULATION ****
info: Entering event queue @ 0.  Starting simulation...
Hello, ARM!
Exiting @ tick 589000 because exiting with last active thread context

Success!

Squelch Annoying LaTeX Output

pdflatex has ridiculously verbose output. Compiling a simple Hello World kind of document is already too much for me.

$ cat hworld.tex
\documentclass[12pt]{article}
\begin{document}
Hello world!
\end{document}

$ pdflatex hworld.tex
This is pdfTeX, Version 3.14159265-2.6-1.40.17
(TeX Live 2016/Debian) (preloaded format=pdflatex)
restricted \write18 enabled.
entering extended mode
(./hworld.tex
LaTeX2e <2017/01/01> patch level 3
Babel <3.9r> and hyphenation patterns for 83 language(s) loaded.
(/usr/share/texlive/texmf-dist/tex/latex/base/article.cls
Document Class: article 2014/09/29 v1.4h Standard LaTeX document class
(/usr/share/texlive/texmf-dist/tex/latex/base/size12.clo))
No file hworld.aux.
[1{/var/lib/texmf/fonts/map/pdftex/updmap/pdftex.map}] (./hworld.aux) )
</usr/sh
are/texlive/texmf-dist/fonts/type1/public/amsfonts/cm/cmr12.pfb>
Output written on hworld.pdf (1 page, 9887 bytes).
Transcript written on hworld.log.

In the specific case of pdflatex, you might try to use the --interaction=batchmode, but (1) this does not squelch all of the output, (2) not all programs have such an option and, most importantly, (3) while the output usually isn't interesting, in case of an error it certainly is.

So I wrote a script called trusting available on Github. It runs a command trusting that it will succeed, squelching stdout and stderr. If errors do occur, i.e. the command exists with a non-zero exit code, the full stdout and stderr output will be printed after that error has occurred.

Show trust in your programs and let them be quiet.

Java Streams for Python Programmers

With Java 8, streams were added to the language. This post describes how typical Python list comprehensions can be implemented in Java using streams.

Creating and Working With Streams

Let's say in Python we have a list l.

>>> l = [1, 5, 1992]

If we wanted to create a list that contains all the squares of the values in l, we would write a list comprehension.

>>> [x**2 for x in l]
[1, 25, 3968064]

We can implement this almost as concise in Java using streams, which live in the java.util.stream.* import. First we need to convert the data at hand into a Stream.

List<Integer> l = Arrays.asList(1, 5, 1992);
Stream<Integer> stream = list.stream();

Now that we have our stream, we want to square up each number. This is a map operation.

Stream<Integer> squares = stream.map(x -> x * x);

The map method actually returns a new stream: A stream that takes the individual values x from the original source and then applies the function x * x to it. But the stream just describes this computation, to actually run it, the stream needs to be consumed. This is similar to how generators work in Python.

We can consume the generated stream piece by piece using the forEach method.

squares.forEach(System.out::println);

In this case, System.out.println is called on every element of the stream. The output is as expected.

1
25
3968064

Instead of using forEach, you can also use iterator method to get, in this case, an Iterator<Integer>.

But we wanted something comparable to list comprehensions. So let's convert the stream to a list.

List<Integer> res = squares.collect(Collectors.toList());

In real code we wouldn't create all these variables of course. A practical version would look something along those lines:

List<Integer> l = Arrays.asList(1, 5, 1992);
List<Integer> res = l.stream()
    .map(x -> x * x)
    .collect(Collectors.toList());

Printing res, we get what we originally asked for.

[1, 25, 3968064]

Filter

Now that we got to know streams, it's time to pick up the pace. Given a list l in Python, if we want only those values that are, say, greater than 100, we can write the following.

>>> l = [1, 5, 15, 515, 15515]
>>> [x for x in l if x > 100]
[515, 15515]

This is called a filter operation because we filter out all those values which do not fit our requirement. It's easy with Java streams.

List<Integer> l = Arrays.asList(1, 5, 15, 515, 15515);
List<Integer> results = l.stream()
    .filter(x -> x > 100)
    .collect(Collectors.toList());

Printing results gives us the expected output.

[515, 15515]

Now what if we want to square only the numbers which are greater than 100? Both in Python and Java this is now easy. In Python you would write a list comprehension.

[x**2 for x in l if x > 100]

In Java we would first run filter and then map on the original stream.

l.stream()
    .filter(x -> x > 100)
    .map(x -> x * x)
    .collect(Collectors.toList());

One, Two, Three

There is one gotcha about streams related to the difference between primitive data type and their boxed counterparts. Let's explore it by example.

In Python it is easy to create a list of integers in a range.

>>> [x for x in range(0, 10)]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

To do the same in Java, we would write something like this.

List<Integer> ls = IntStream.range(0, 10)
    .boxed()
    .collect(Collectors.toList());

Printing ls we get the expected.

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Note the call to the boxed method. It converts the IntStream, a stream of int values to Stream<Integer>, a stream of Integer values. This is required because there can be no List<int> in Java, only a List<Integer>. Similar to this, there is DoubleStream which serves the same purpose for floating point numbers.

We can also move the other way, that is from Stream<Integer> to IntStream, indeed from Stream<MyCoolType> to IntStream using the mapToInt method of Stream. We might want to do this because IntStream has cool methods like sum which Stream lacks. Consider the following code in which we sum up the price of multiple products in a shopping cart. In Python we would write something like this.

Product = namedtuple('Product', ('name', 'price'))

def totalPrice(cart: List[Product]):
    return sum(p.price for p in cart)

We can write similar code in Java now.

class Product {
    String name;
    int price;
}

int totalPrice(List<Product> cart) {
    return cart.stream()
        .mapToInt(p -> p.price)
        .sum();
}

Resume

We can now translate Python list comprehensions to Java using streams. I didn't show it here, but this leads to shorter and cleaner code compared to using traditional loops.

I do have to mention that streams are way more powerful than what was shown here. Interesting features include advanced reduction (think sum on steroids) and parallel execution. If you are interested, consider reading this Java 8 Stream Tutorial or if you are feeling courageous start at the official documentation.