Cross-compiling x264 for win32 on Ubuntu Linux
The total lack of documentation on compiling x264 (and dependencies) for win32 on a linux32 system is henceforth rectified. This guide assumes you are using Ubuntu 9.10 and the packaged version of mingw32. Newer versions of the below packages might require additional/less wrangling.
Required packages in the base system:
sudo apt-get install pkg-config yasm subversion cvs git-core mingw32
Create the basic tree for installing win32-compatible dependancies to:
mkdir -p ~/win32-x264/{src,lib,include,share,bin}
Place this helper script at ~/win32-x264/mingw and chmod +x it:
#!/bin/sh export CC=i586-mingw32msvc-gcc export CXX=i586-mingw32msvc-g++ export CPP=i586-mingw32msvc-cpp export AR=i586-mingw32msvc-ar export RANLIB=i586-mingw32msvc-ranlib export ADD2LINE=i586-mingw32msvc-addr2line export AS=i586-mingw32msvc-as export LD=i586-mingw32msvc-ld export NM=i586-mingw32msvc-nm export STRIP=i586-mingw32msvc-strip export PATH="/usr/i586-mingw32msvc/bin:$PATH" export PKG_CONFIG_PATH="$HOME/win32-x264/lib/pkgconfig/" exec "$@"
Now to install pthread & zlib:
cd ~/win32-x264/src wget -qO - ftp://sourceware.org/pub/pthreads-win32/pthreads-w32-2-8-0-release.tar.gz | tar xzvf - cd pthreads-w32-2-8-0-release make GC-static CROSS=i586-mingw32msvc- cp libpthreadGC2.a ../../lib cp *.h ../../include
cd ~/win32-x264/src wget -qO - http://zlib.net/zlib-1.2.4.tar.gz | tar xzvf - cd zlib-1.2.4 ../../mingw ./configure # Remove references to "-lc" from the Makefile (tells GCC to link output with libc, which is implied anyway, and explicit declaration causes a script error) sed -i"" -e 's/-lc//' Makefile make DESTDIR=../.. make install prefix=
Installing FFmpeg:
cd ~/win32-x264/src svn checkout svn://svn.ffmpeg.org/ffmpeg/trunk ffmpeg cd ffmpeg # Delete references to -Wmissing-prototypes, a GCC warning that fails when cross-compiling sed -i"" -e '/missing-prototypes/d' configure ./configure \ --target-os=mingw32 --cross-prefix=i586-mingw32msvc- --arch=x86 --prefix=../.. \ --enable-memalign-hack --enable-gpl --enable-avisynth --enable-postproc --enable-runtime-cpudetect \ --disable-encoders --disable-muxers --disable-network --disable-devices make make install
Installing FFmpegsource:
cd ~/win32-x264/src svn checkout http://ffmpegsource.googlecode.com/svn/trunk/ ffms cd ffms ../../mingw ./configure --host=mingw32 --with-zlib=../.. --prefix=$HOME/win32-x264 ../../mingw make make install
Installing GPAC:
Special thanks to the GPAC dev who kindly assisted me in beating the terrible configure/Makefile scripts into shape.
cd $HOME/win32-x264/src # Create a CVS auth file on your machine cvs -d:pserver:anonymous@gpac.cvs.sourceforge.net:/cvsroot/gpac login cvs -z3 -d:pserver:anonymous@gpac.cvs.sourceforge.net:/cvsroot/gpac co -P gpac cd gpac chmod +rwx configure src/Makefile # Hardcode cross-prefix sed -i'' -e 's/cross_prefix=""/cross_prefix="i586-mingw32msvc-"/' configure ../../mingw ./configure --static --use-js=no --use-ft=no --use-jpeg=no --use-png=no --use-faad=no --use-mad=no --use-xvid=no --use-ffmpeg=no --use-ogg=no --use-vorbis=no --use-theora=no --use-openjpeg=no --disable-ssl --disable-opengl --disable-wx --disable-oss-audio --disable-x11-shm --disable-x11-xv --disable-fragments--use-a52=no --disable-xmlrpc --disable-dvb --disable-alsa --static-mp4box --extra-cflags="-I$HOME/win32-x264/include -I/usr/i586-mingw32msvc/include" --extra-ldflags="-L$HOME/win32-x264/lib -L/usr/i586-mingw32msvc/lib" # Fix pthread lib name sed -i"" -e 's/pthread/pthreadGC2/' config.mak # Add extra libs that are required but not included sed -i"" -e 's/-lpthreadGC2/-lpthreadGC2 -lwinmm -lwsock32 -lopengl32 -lglu32/' config.mak make # Make will fail a few commands after building libgpac_static.a (i586-mingw32msvc-ar cr ../bin/gcc/libgpac_static.a ...). That's fine, we just need libgpac_static.a cp bin/gcc/libgpac_static.a ../../lib/ cp -r include/gpac ../../include/
Building x264:
cd ~/win32-x264/src git clone git://git.videolan.org/x264.git cd x264 ./configure --cross-prefix=i586-mingw32msvc- --host=i586-pc-mingw32 --extra-cflags="-I$HOME/win32-x264/include" --extra-ldflags="-L$HOME/win32-x264/lib" make
Leave to cool for 15 minutes. Serves four.
Changelog:
- 20100518: Updated ffmpeg configure args. ffms build needs mingw wrapper. Add cvs to required packages.
Tarsnap backups on Windows and Linux
For the past month or so I've been running Tarsnap to backup my home desktop and remote servers. Tarsnap is an online (payware) backup system that's written by Colin Percival, and is probably the most technically impressive backup system I've seen. As well as that, it's damn cheap.
What's so great about Tarsnap? Here's three things:
- Snapshot backups. Every backup you create with tarsnap is a standalone 'snapshot' of your data, totally independent of all other snapshots.
- Backup deduplication. So your backup includes a 300mb log file that gets a few megs of data written to it every day. Tarsnap recognises this, and for each new snapshot only the changed data has to be uploaded and stored.
- Crazy Ass Security. While mild-mannered Colin Percival works on Tarsnap during the day, by night he's the FreeBSD Security Officer. He's been there since 2005, so must be doing something right. Plus, check out this page. How can something with that many mentions of AES, SHA and RSA be insecure?! :-)
This post isn't a HOWTO on setting up Tarsnap, there's a comprehensive tutorial on the subject already. This post is just to document how I use Tarsnap, on both Linux and Windows.
Linux
My Linux servers perform a simple daily backup of everything in certain folders. Technically, I could simply backup "/" and exclude the directories I don't want, but smaller archives are faster to restore from, and there's no monetary penalty for having more snapshots. This is my script:
#!/bin/bash for dir in $(cat /root/tarsnap-dirs) ; do nice tarsnap -c -f $(hostname -s)-$(date -u +%Y%m%d-%H%M%S)-$(echo $dir | tr -d '/') --one-file-system -C / $dir done # Delete backups more than n days old # n=10 # tarsnap --list-archives | sort | cut -d- -f1-2 | uniq | tail -n +$n > /tmp/temp.$$ # tarsnap --list-archives | fgrep -f /tmp/temp.$$ | while read archive ; do # echo Deleting $archive # tarsnap -d -f $archive # done # rm /tmp/temp.$$ tarsnap --print-stats
(This script runs from crontab, so the output gets mailed to me daily. The email is sent to the address specified as the MAILTO variable in /etc/crontab.)
My .tarsnaprc looks like this:
keyfile /root/tarsnap-key-abraxo.key cachedir /root/tarsnap-cache/ exclude /root/tarsnap-cache/ humanize-numbers
Each day, a backup of each folder listed in the file 'tarsnap-dirs' is created, with names like: 'bluebottle-20091209-200001-homeaj'. There's commented out support for deleting old archives too, but my monthly costs are so low I keep everything.
Windows
My Windows setup is basically identical, but since there's no native Windows (or msys) support for Tarsnap, you have to make do with Cygwin. Explaining how to install Cygwin is far beyond the scope of this document, but it's pretty simple. Apart from the standard Tarsnap dependancies, you will probably also want to install 'ssmtp', which will let you email Tarsnap's output to yourself like Unix cron does.
(You can generate ssmtp's config file by running 'ssmtp-config'.)
Again, my script:
#!/bin/bash log=/tmp/tarsnap.log.$$ cat <$log From: Alex Jurkiewicz Subject: tarsnap run $(hostname)-$(date +%Y%m%d) To: Alex Jurkiewicz EOF tarsnap -c -f $(hostname)-$(date +%Y%m%d-%H%M%S)-homeaj -C /home/ aj >>$log 2>&1 tarsnap -c -f $(hostname)-$(date +%Y%m%d-%H%M%S)-CUsersAlex -C /cygdrive/c/Users/ Alex >>$log 2>&1 tarsnap --print-stats >>$log 2>&1 cat $log | /usr/sbin/ssmtp.exe alex@bluebottle.net.au rm $log
Because I want to exclude a lot of directories on Windows, I put these in my ~/.tarsnaprc file:
exclude Desktop/ exclude AppData/Local/Temp/
And so on. So there you have it. Simple, painless backups with my favourite new toy, Tarsnap.
Comment (0)Ubuntu 9.10 nvidia-settings - xorg.conf save fail
Ubuntu 9.10 creates a (relatively) minimal xorg.conf file compared to what older Ubuntus have. This causes the nvidia-settings app to be unable to parse the file (and therefore save a new version), specifically because of this missing line from the "Screen" section:
Device "Default Device"
If you add this line to your default screen configuration in /etc/X11/xorg.conf, nvidia-settings should be happy again.
Gripe: didn't anyone pick this up during testing?
Comment (1)Scripted FreeBSD reimaging
Recently at work I've been working on a system to automatically revert FreeBSD systems to a known good state - something like a VM snapshot, but for physical machines too. The system is reasonably fast, portable across different hardware configurations and very easy to use. It's turned out that this system has worked quite well, and although reasonably simple is the result of much research and time, so I'm posting some pointers here.
The Problem
The push for this came from a need to improve the testing environments our developers have been using. The old testing environments were VMs, more or less handmade at some point in the distant past to provide a rough approximation of our live environment. Since that time the developers had mostly looked after them themselves, tweaking, fixing and forking the VMs as they saw fit. New developers would copy an existing developer's VM, and branch off from there.
This system was bad. There were problems ranging from minor version differences all the way up to entirely missing subsystems, not to mention the total lack of documentation. Every developer changed their VM slightly differently, so troubleshooting problems was as much exploration and discovery as debugging. Worse, the slowly growing number of environments made keeping them in sync (or something approximating that) with the live environment increasingly difficult. Something had to be done.
The Solution
What I've ended up going with is a very basic system that builds on as much of our previous infrastructure as possible. The key component is our 'live environment image', an OS image that we base all our live servers off. The new development environments are also based off this image, and the solution consists of a set of scripts that automate the conversion of the live environment image into a development environment. So that's the high-level concept, how do the nitty gritty details work?
The system is made up of two separate installs of FreeBSD on one disk. The first install is a 5gb disk slice, containing a minimal install of FreeBSD. This is the Reimaging OS. The second install is on another slice taking up the rest of the disk and is the actual Development Environment.
The Reimaging OS
This copy of FreeBSD isn't just a fresh install. There are a few changes, the most important are the installation of bash and these two lines added to the machine's /etc/rc.local file, which start the magic:
scp -i /root/key reimageuser@fileserver:dev-environment-reimage.sh /tmp/ /usr/local/bin/bash /tmp/dev-environment-reimage.sh
And what does this script do? Well, in short:
- Format the partitions making up the Development Environment.
- Restore the base 'live environment image' into the Development Environment.
- Modify the fresh Development Environment to actually be suitable for use as a Development Environment, rather than a live server.
- Change the default boot slice to the second slice, and reboot the machine.
So now the machine reboots, and when the bootloader starts up, it loads...
The Development Environment
Remember step three above? The hand-wavey "turn the live environment into a dev. environment" step? Well, you can't fully complete that step from the Reimaging OS. There are some things you just don't know. What hostname should you set for the machine? What email address should all outgoing mail be redirected to? It turns out that the first time a dev. environment boots, the developer has to answer some questions. After developers reimage their machine, the final step is to run a script on first login: dev-environment-rechristen.sh. This script goes through and makes all the changes to the system that require user input, plus a few workplace specific changes.
Of course, the developer also needs to be able to kick off a reimage of their environment at some point in the future. How do they do that? Another script sets the default boot OS to the Reimaging OS and reboots the machine. Developer goes away for a coffee and comes back to a fresh machine.
The Scripts
So that's how it works, how about some sample code? Please note that these are edited versions of the scripts I run with anything even vaguely revealing about our configuration/infrastructure stripped out. They're a starting point for you, but you still need to do a fair amount of work to get these scripts working for you.
- reimage-setup.txt. This is a short script I wrote to help quickly deploy the Reimaging OS to new machines. Put it on a thumbdrive with a FreeBSD dump(8) image, then boot off a FreBSD install CD, and select Fixit -> Live CD Filesystem. Mount the USB key at /mnt (something like mount /dev/da0s1 /mnt) and then run the script like so: /mnt/reimage-setup.sh /dev/ad0 (where /dev/ad0 is the drive you want to use for the dev. environment. The script (or something like it) has worked well for me across a number of heterogeneous systems, but, of course, YMMV. For the record, the rest of these scripts are written in bash, so make sure that's available in your OSes.
- reimage-do.txt. This script is the basic skeleton of our dev-environment-reimage.sh script. It shares many of the assumptions the previous script makes: what the partitions are, where they are located and so on. I've excised the more site-specific customisation, but it's ready to go otherwise.
- reimage-begin.txt. This script completes the trio. It's what you run from within the development environment to kick off the reimaging process. Nice and simple, and again shares all the assumptions the previous two scripts made.
- Bonus: a starting point for your reimage-rechristen.sh script. A couple of functions that ours does. Most of the rest of our script does site-specific work, like checking some files out of an SVN repository, changing the hostname in a few config files, and so on.