SBR600: Release 0.3, rebuilding X Server package

As a recap of the results of my 0.2 release, tweaking xorg.conf did not have a huge impact on reducing memory usage. Chris suggested I should try from another direction which is to repackage the X Server. Due to time constraints and the limited availability of the Pi, I decided to repackage the X Server in an x86 environment inside my VMware Workstation virtual machine that I used in my 0.1 release. Following the steps I learned in SBR600, I installed and recreated the necessary RPM build directories. While I looked inside the X Server source package, it seems that there were a lot of unneeded packages that were included such as Xnest or devel. I generated the SPEC file that I was going to modify using:

rpm -ivh sourcepackage-name*.src.rpm

Finally, I rebuilt the package with my modified SPEC file and tested for errors using rpmlint. Rpmlint showed no errors so I reinstalled this new package and tested for the memory usage again. Memory usage showed insignificant changes. The default X Server in my virtual machine was using around 38mb of memory. The modified X Server I installed showed around 37mb of memory used.

However, it seems that the X Server on the x86 platform used less memory compared to the ARM on the Raspberry Pi which was around 43mb. I can two reasons for my results in this release. Either the changes I made when I rebuilt the X Server package were not important in reducing memory usage or the X Server generally requires that much memory.

I have included my SRPM, RPM, common files used and SPEC file for others to take a look at. I hope others will have more success than me.

The PowerPoint I used during my presentation can be found here. I’ve also updated changes I made in my original project plan for my 3 releases here.

SBR600: Release 0.2, creating xorg.conf and testing on the Pi

In my previous release, I tested how much memory XWindows was using in a virtualized environment running Fedora 13. This time, I first tweaked xorg.conf in a VM. After talking to Chris, I found out that there were some extensions that could be turned off to save memory usage. After running some tests on disabling these extensions, I proceeded to replicate my process on the Raspberry Pi.

Initially, I had troubles starting the Pi with the X Server. I found out that the Pi could boot into a GUI mode by eliminating some layers inside the X Windows.

Normally, I could generate an xorg.conf template with the command:

X –configure :1

However, this would crash the Pi so I could not build a template to use. In the end, I built an xorg.conf from scratch, including only sections that I want to configure. After X7 windows, the developers made it so that X would probe for values needed to configure X Windows. Therefore, the sections that were missing from my configuration would be filled in automatically.

I ran this command to open up just a terminal window in X to see how much memory was being used:

X :0 -ac & sleep 2; xterm -display :0 & metacity

Around 36mb memory was used.

I found out what extensions were enabled on the Pi after I started X Windows using:

Xdpyinfo

This generated an output of what X was using. I’ve included a part of the output below:

name of display:    :0.0
version number:    11.0
vendor string:    Fedora Project
vendor release number:    10802000
maximum request size:  16777212 bytes
motion buffer size:  256
bitmap unit, bit order, padding:    32, LSBFirst, 32
image byte order:    LSBFirst
number of supported pixmap formats:    7
supported pixmap formats:
depth 1, bits_per_pixel 1, scanline_pad 32
depth 4, bits_per_pixel 8, scanline_pad 32
depth 8, bits_per_pixel 8, scanline_pad 32
depth 15, bits_per_pixel 16, scanline_pad 32
depth 16, bits_per_pixel 16, scanline_pad 32
depth 24, bits_per_pixel 32, scanline_pad 32
depth 32, bits_per_pixel 32, scanline_pad 32
keycode range:    minimum 8, maximum 255
focus:  window 0x1000004, revert to Parent
number of extensions:    26
BIG-REQUESTS
Composite
DAMAGE
DOUBLE-BUFFER
DPMS
DRI2
GLX
Generic Event Extension
MIT-SCREEN-SAVER
MIT-SHM
RANDR
RECORD
RENDER
SGI-GLX
SHAPE
SYNC
X-Resource
XC-MISC
XFIXES
XFree86-DGA
XFree86-VidModeExtension
XINERAMA
XInputExtension
XKEYBOARD
XTEST
XVideo
default screen number:    0
number of screens:    1

screen #0:
dimensions:    1920×1080 pixels (508×286 millimeters)
resolution:    96×96 dots per inch
depths (7):    16, 1, 4, 8, 15, 24, 32
root window id:    0xe3
depth of root window:    16 planes
number of colormaps:    minimum 1, maximum 1
default colormap:    0x20
default number of colormap cells:    64
preallocated pixels:    black 0, white 65535
options:    backing-store NO, save-unders NO
largest cursor:    1920×1080

The output showed that the Pi used 26 extensions. Running top command, the default X Windows used around 43mb of memory and 5% of CPU when idle with one terminal window open. The CPU would spike to around 50 to 60% if I frantically dragged the terminal window around. I used this as a base for comparing the results after I changed my configuration.

After researching on what the extensions are used for, there are several that seemed unnecessary for the Pi.

MIT-SCREEN-SAVER

This is used to inform screensavers when to start and stop.

RANDR

This stands for rotate and resize. Useful for display setting changes. Not useful for the Pi because it will be connected to a 1080p screen.

RECORD

This is used for application testing as it records X events.

SHAPE

This enables nonrectangular windows. It can be disabled if the Pi only has rectangular windows.

XInputExtension

This provides support for specialized input devices like tablets etc.

XTEST

Used for performance benchmarking.

XINERAMA

Used for single screen, multimonitor support.

XVideo

Enables video streams (ie. from video camera) to be converted, transformed on X. Improves video performance.

After turning off these extensions, I started X and ran ‘top’ to see memory usage. It still idled at 42mb memory used and 5% CPU used. This was surprising because I initially thought that this would decrease memory usage.

As a final test, I wanted to see what would happen if I tried to disable all the extensions. Here was the configuration I made to disable all the extensions.

Section “Screen”
Identifier “Screen0”
Device “Card0”
Monitor “Monitor0”
Subsection “Display”
Depth 24
Modes “1920×1080”
EndSubsection
Subsection “Display”
Depth 32
Modes “1920×1080”
EndSubsection
EndSection

Section “Extensions”
Option “BIG-REQUESTS” “off”
Option “Composite” “off”
Option “DAMAGE” “off”
Option “DOUBLE-BUFFER” “off”
Option “DPMS” “off”
Option “DRI2” “off”
Option “GLX” “off”
Option “Generic Event Extension” “off”
Option “MIT-SCREEN-SAVER” “off”
Option “MIT-SHM” “off”
Option “RANDR” “off”
Option “RECORD” “off”
Option “RENDER” “off”
Option “SGI-GLX” “off”
Option “SHAPE” “off”
Option “SYNC” “off”
Option “X-Resource” “off”
Option “XC-MISC” “off”
Option “XFIXES” “off”
Option “XFree86-DGA” “off”
Option “XFree86-VidModeExtension” “off”
Option “XINERAMA”
Option “XInputExtension” “off”
Option “XKEYBOARD” “off”
Option “XTEST” “off”
Option “XVideo” “off
EndSection

I started X and ran ‘xdpyinfo’ again.

name of display:    :0.0
version number:    11.0
vendor string:    Fedora Project
vendor release number:    10802000
maximum request size:  16777212 bytes
motion buffer size:  256
bitmap unit, bit order, padding:    32, LSBFirst, 32
image byte order:    LSBFirst
number of supported pixmap formats:    7
supported pixmap formats:
    depth 1, bits_per_pixel 1, scanline_pad 32
    depth 4, bits_per_pixel 8, scanline_pad 32
    depth 8, bits_per_pixel 8, scanline_pad 32
    depth 15, bits_per_pixel 16, scanline_pad 32
    depth 16, bits_per_pixel 16, scanline_pad 32
    depth 24, bits_per_pixel 32, scanline_pad 32
    depth 32, bits_per_pixel 32, scanline_pad 32
keycode range:    minimum 8, maximum 255
focus:  window 0x1000004, revert to Parent
number of extensions:    8
    BIG-REQUESTS
    DRI2
    Generic Event Extension
    SHAPE
    SYNC
    XC-MISC
    XInputExtension
    XKEYBOARD
default screen number:    0
number of screens:    1

Some extensions were still used. These were probably left on because they were needed for X to function properly. I ran the ‘top’ command and it still used around 42mb of memory and 5% CPU when idle. An interesting observation was that when I tried to drag and open new windows, the CPU would go up to 70%. The changes I made actually increased CPU usage. This was probably because X Windows had to redraw the windows that were overlaid which the CPU does.

In conclusion, the disabling extensions didn’t really help save memory usage. I would try to test out specific functions that pertain to each extension to see what happens as a result. As of now, I have only tested with terminal windows.

SBR600: Release 0.1, testing xorg.conf in virtual machines

The goal of my release 0.1 for the project was to successfully have X Windows running in a virtualized environment with similar hardware specs to the Raspberry Pi board. I wanted to use VM’s for testing because there was only one Raspberry Pi board that needed to be shared around for 5 different projects. The other reason was that I can freely test different settings with ease as I can recreate the VM if I needed to. I will also be installing the VM in two different environments.

1. Running Virtual Box, Fedora 13 x86
2. Running VMware Workstation, Fedora 13 x86

Installation of the OS:

Initially, I built a Fedora 13 x86 (Fedora-13-i686-Live.iso) machine with Virtual Box inside my Ubuntu 11 host.  With only 256mb RAM and 12mb of video memory, the machine did install and boot up successfully. Later on, I lowered the RAM settings and repeated the testing process. This was again repeated on VMware workstation.

Checking System Resources:

 The first thing I checked was how much system memory it was using. According to the system monitor, this machine used 125mb out of 245mb of usable memory.

 Generating Xorg.conf

After some research, I found that X11/Xorg that now comes on Fedora 13 doesn’t store a default xorg.conf file anymore. X11 can run without any xorg.conf file. Instead, it uses probed/default values that are generated during startup. To generate a template xorg.conf, I’ve installed system-config-display. I’ve mainly used default values for testing purposes starting with an 800×600 resolution and the standard drivers used for the VM.

Test Cases:

 Virtual Box VM:

With 256mb RAM:
I’ve set the default resolution to 800×600 and the system memory hovered at around 128/245mb. The other resolution I tested was at 1920×1080 and the memory was around 135/245mb.

With 192mb RAM:
800×600, 109/182mb
1920×1080, 120/182mb

With 128mb RAM:
800×600, 78/119mb
1920×1080, 93/182mb

VMware Workstation VM:

With 256mb RAM:
800×600, 124/245mb
1920×1080, 132/245mb

With 192mb RAM:
800×600, 106/182mb
1920×1080, 116/182mb

With 128mb RAM:
800×600, 66/119mb
1920×1080, 81/119mb

Summary:

VirtualBox:

	800×600		1920×1080
RAM allocated	RAM used
245mb RAM	128mb	52.2%	135mb	52.2%
182mb RAM	109mb	60.0%	120mb	65.9%
119mb RAM	78mb	65.5%	93mb	78.1%

VMware Workstation:

	800×600		1920×1080
RAM allocated	RAM used
245mb RAM	124mb	50.6%	132mb	53.9%
182mb RAM	106mb	58.2%	116mb	63.7%
119mb RAM	81mb	68.1%	81mb	68.1%

Conclusion:

A general pattern can be concluded from these two environments. The Fedora machine in VMware workstation used less resources overall on all test cases compared to the VirtualBox companion. As the system memory decreases, a larger proportion of memory was being used. This is true for both environments. As the memory decreases, start up times increased and the machines became less responsive as predicted. With 256mb memory, the system was running quite decently. 128mb memory was too sluggish and not very user friendly. Although this is being tested in a VM with an x86 architecture, test results on the actual Raspberry Pi may vary. A possible goal for further tweaking for the second release would be toning down the system memory used to below 60%.

Creating a Repository-release RPM

Signing RPM packages is an important step because it ensures the authenticity of the file that is being installed on a system.

I’ve created my own key using the command:

gpg –gen-key

I’ve also added my email address to the key by adding in ‘%_gpg_name clchor@learn.senecac.on.ca’ in my ~/rpmmacros file. I’ve used the package I created previously to put in my repository and signed it with my key using:

rpm –addsign packagefile

Created the repository metadata using:

createrepo .

After I copied a sample .repo file from /etc/pki/rpm-gpg/ to the directory that has my signed rpm package. I’ve made edits to the .repo file and added my own gpgkey named ‘clchor-gpg-key’.

Here are the files inside the repos:

repo/
|– clchor-gpg-key
|– hello-2.7-99.x86_64.rpm
`– repodata
    |– filelists.xml.gz
    |– other.xml.gz
    |– primary.xml.gz
    `– repomd.xml

1 directory, 6 files

I have created a tarball for the above located here.

Unfortunately, I have not been successful at installing this from my repository. I must have missed some crucial step during the process. If someone can comment how what I’m doing wrong, it would be much appreciated.

Using Mock and Koji

After successfully building my SRPM from the previous post, I tested the buildrequires with mock.

    yum install mock
    usermod -G mock clchor
    mock -r fedora-14-x86_64 yourSRPM

This took several minutes as I was using a virtual environment. Fortunately, it ran without issues.

The next step was to test the build on another system using Koji. I was following the instructions here: http://fedoraproject.org/wiki/PackageMaintainers/UsingKoji

To summarize the steps:

    yum install fedora-packager
    /usr/bin/fedora-packager-setup

This would prompt you for you FAS account credentials and passphrase for your certificates. After generating the certificates, follow the instructions listed in the above to add the certificate to your firefox browser.

After getting the certificates set up, I proceeded to have koji to use my SRPM to build in a f14 environment. Here is the trouble I ran into last night.

I was running this command under my user account ‘clchor’:

    koji build dist-f14 –scratch yourSRPM

It kept failing with an error message like this:

Error: [(‘SSL routines’, ‘SSL3_READ_BYTES’, ‘sslv3 alert certificate revoked’), (‘SSL routines’, ‘SSL3_READ_BYTES’, ‘ssl handshake failure’)]

I kept thinking that there was something wrong with my certificates and asked people on #seneca channel. They thought the same thing. I even tried to manually download the certificates following the instructions here (http://fedoraproject.org/wiki/PackageMaintainers/UsingKoji). Apparently, this command works only works in my root user account which is quite interesting because the error message made no note of that.

After running the koji command successfully, I can see the build process start on my terminal as well as http://koji.fedoraproject.org/koji

This process whole process of running it using koji was only a matter of minutes as the package I used was quite small amd simple.

Initially, I thought that the process was simple and easy to use. But after having the nightmare of getting Koji to work properly, I would like to think otherwise. It was fun talking to some folks on the #seneca channel as well as trying to troubleshoot my steps. I remembered Chris saying not to post large pieces of commands and errors directly in the channel and to use fpaste.org. Good thing I remembered because I’m sure someone would’ve yelled at me.

RPM Writing

In this post, I will illustrate the steps I took in order to build a RPM with my own modified spec file.

First I installed the necessary packages:

    yum groupinstall “Fedora Packager”
    yum install rpmlint yum-utils

Under my user account, I ran this command to set up my rpmbuild directories:

    rpmdev-setuptree

I copied my tarball (using http://mirror.csclub.uwaterloo.ca/gnu/hello/hello-2.7.tar.gz) from my previous post to ~/rpmbuild/SOURCES.

Changed to ~/rpmbuild/SPECS and created a new skeleton spec file using:

   rpmdev-newspec hello.spec

I’ve edit the specfile to my own requirements as shown below.

Name:           hello
Version:        2.7
Release:        99
Summary:        Cho Long (Tommy) Chor’s Hello World program

License:    GPLv3+
URL:        http://www.clchor.com
Source0:    hello-2.7.tar.gz

BuildRequires: gettext
# Requires:       
Requires(post): info
Requires(preun): info

%description
This is a test “Hello World” RPM package created by a student. Please advise before installing.

%prep
%setup -q

%build
%configure
make %{?_smp_mflags}

%install

make install DESTDIR=%{buildroot}
%find_lang %{name}
rm -f %{buildroot}/%{_infodir}/dir

%post
/sbin/install-info %{_infodir}/%{name}.info %{_infodir}/dir || :

%preun
if [ $1 = 0 ] ; then
/sbin/install-info –delete %{_infodir}/%{name}.info %{_infodir}/dir || :
fi

%files -f %{name}.lang
%{_bindir}/hello
%doc %{_mandir}/man1/hello.1.gz
%doc %{_infodir}/%{name}.info.gz

%changelog
* Wed Oct 4 2011 Hello World package modified by Cho Long (Tommy) Chor 2.7-99
– Initial release for testing purposes

To build my package using my spec file, run:

    rpmbuild -ba nameOfPackage.spec

Getting the above step to work was the hardest part. Initially, I had errors with missing or wrong sections like having the wrong Source0 field. And then the build proceeded into later sections, the variables I used were wrong. But after much effort, the build was successful and the binary RPM and source RPM was created.

To further test the RPM, I ran rpmlint on the binary and source RPM and was error free.

Building from Source

I’ve gone through several packages. Some were big while others are not supported anymore like gnuchess. The ones I’ve used for this demonstration is “hello” and spell”.

I’ve included the download links below:

http://mirror.csclub.uwaterloo.ca/gnu/hello/hello-2.7.tar.gz
http://ftp.gnu.org/gnu/spell/spell-1.1.tar.gz

First I installed the necessary tools using the command:

yum groupinstall “Development Tools” “Development Libraries”

I used wget to download the two tarballs and extracted it using:

tar xvzf <package name>

For each extracted tarball, a directory of files is created. Within the directory, run the following commands to compile and install the package.

./configure; make; make install

I’ve used the time command to see how long each process takes.

Hello:
time ./configure

real    0m41.788s
user    0m18.854s
sys    0m22.369s

time make

real    0m2.328s
user    0m1.358s
sys    0m0.952s

time make install

real    0m2.582s
user    0m1.001s
sys    0m1.678s

Spell:
time ./configure

real    0m6.741s
user    0m3.092s
sys    0m3.929s

time make

real    0m0.377s
user    0m0.253s
sys    0m0.090s

time make install

real    0m0.371s
user    0m0.218s
sys    0m0.114s

Please take note that this is hosted in a virtualized environment with 2 processors and 2GB of RAM so the process does take longer than expected.

Introduction

Hi my name is Tommy (Cho Long) Chor and I am taking SBR600 this semester.

For my final project, I will be working on the Raspberry Pi trying to optimize the X configuration.

I’ve been trying to troubleshoot a problem with Koji and trying to authenticate with the certificates. Below is a part of the conversation in #seneca on freenode.net

toomy> I’m working on the koji lab for SBR600, but whenever I run “koji build dist-f14 –scratch <pkg name>.rpm” I get an error
<dayuster> I am also having this issue
<toomy> I’ve included my output in here: http://fpaste.org/GjP8
<mhoye> Is that your machine, or a lab machine?
<toomy> my machine, in a VM
<toomy> I followed this: http://fedoraproject.org/wiki/PackageMaintainers/UsingKoji to create my certs
<mhoye> Ok, so, in your home directory, how many certs do you have?
* dayuster has quit (Quit: Leaving)
<toomy> one, it’s named “fedora-browser-cert.p12”
<toomy> I’ve added that in my firefox browser
<mhoye> hmmm.
<mhoye> Try an ls -A rather than an .ls
<mhoye> an “ls”, rather.
<mhoye> Adding it to firefox is not a thing, I don’t think.
<mhoye> According to that UsingKoji, there should be three keys there, called .fedora.cert .fedora-upload-ca.cert and .fedora-server-ca.cert
<mhoye> Make sure they’re all there; there are links to download them in the document.
<mhoye> Basically: this is an SSL problem, which probably means you’ve overlooked a step involving the certificates.

Here is a link to my Seneca wiki page: http://zenit.senecac.on.ca/wiki/index.php/User:Clchor

Here is a link to my fedora wiki page: https://fedoraproject.org/wiki/User:Clchor