In depth: A lot of people have been chattering
about the improvements Windows 7 brings for Windows users, but how does
it compare to Ubuntu in real-world tests? We put Ubuntu 8.10, Windows
Vista and Windows 7 through their paces in both 32-bit and 64-bit tests
to see just how well Ubuntu faces the new contender. And, just for
luck, we threw in a few tests using Jaunty Jackalope with ext4.
When Windows users say that Windows 7 is easier to install than
ever, what do they really mean? When they say it's faster, is it just
in their heads, or is Microsoft really making big strides forward? And,
perhaps most importantly, when Linux benchmarkers show us how
screamingly fast ext4 is compared to ext3, how well do those figures
actually transfer to end users?
These are the questions we wanted to answer, so we asked Dell to
provide us with a high-spec machine to give all the operating systems
room to perform to their max. Our test machine packed an Intel Core i7
920, which in layman's terms has four cores running at 2.67GHz with
hyperthreading and 8MB of L3 cache. It also had 6GB of RAM, plus two
500GB of hard drives with 16MB of cache.
The tests we wanted to perform for each operating system were:
- How long does each operating system take to install?
- How much disk space was used in the standard install?
- How long does boot up and shutdown take?
- How long does it take to copy files from USB to HD, and from HD to HD?
- How fast can it execute the Richards benchmark?
We also, just for the heck of it, kept track of how many mouse clicks it took to install each OS.
Before we jump into the results, there are a few things we should make clear:
- To ensure absolute fairness, install time was measured from the
moment the computer was turned on until we reached a working desktop.
- The same computer hardware was used for all tests, and all operating systems were installed fresh for this article.
- We used the Ultimate versions of Windows Vista and Windows 7, simply because Windows 7 was provided only in this flavour.
- We used the Windows Vista SP1 disk to accurately reflect what users are likely to experience todaay.
- Our Windows 7 version is the open beta that Microsoft issued
recently. It is probable Windows 7 will be at least this fast in the
final build, if not faster.
- For Ubuntu 9.04 we used the daily build from January 22nd.
- All operating systems were installed using standard options; nothing was changed.
- After checking how much space was used during the initial install,
each operating system was updated with all available patches before any
other tests were performed.
- Our journalistic friends have informed us that Windows Vista (and,
presumably, Windows 7 too) has technology to increase the speed of the
system over time as it learns to cache programs intelligently. It also
allows users to use flash drives to act as temporary storage to boost
speed further. None of our tests are likely to show this technology in
action, so please take that into account when reading the results.
- The filesystem, boot, shutdown and Richards benchmarks were performed three times each then averaged.
And, of course, there's the most important proviso of all: it is
very, very likely that a few tweaks to any of these operating systems
could have made a big difference to these results, but we're not too
interested in that - these results reflect what you get you install a
plain vanilla OS, like most users do.
Install time
Amount of time taken to install, from machine being turned on to working desktop. Measured in seconds; less is better.
At first glance, you might think that Ubuntu clearly installs far
faster than either version of Windows, and while that's true there is
one important mitigation: both Windows Vista and Windows 7 run system
benchmarks part-way through the installation to determine the
computer's capabilities.
A bit of a flippant one - just how many mouse clicks does it take to install an OS with the default options?
Surprisingly, Ubuntu 8.10 gets it done with half the clicks of
Windows 7. NB: hopefully it's clear this doesn't make Ubuntu 8.04 twice
as easy to install. Measured in, er, mouse clicks; fewer is better.
Disk space used immediately after a fresh install. Measured in gigabytes; less is better.
While some people might complain that we used the Ultimate editions
of both Vista and Windows 7, they probably forget that the standard
Ubuntu includes software such as an office suite as standard. NB: Vista
failed to detect the network card during install, leaving us without an
internet connection until a driver was downloaded on another computer.
Bootup and shutdown
Boot up time was also measured from the moment the machine was
turned on, and the timer was stopped as soon as the desktop was
reached. The Dell box does take about 20 seconds to get past POST, but
to avoid questions about when to start the timer we just started it as
soon as the power button was pressed.
Amount of time taken to boot, from machine being turned on to working desktop. Measured in seconds; less is better.
The 32-bit version of Windows 7 is the only one to beat the
one-minute mark, but that advantage is quickly lost in the switch to
64-bit. Linux has always been rather slow to boot, but as we understand
it reducing boot time is one of the goals of the Ubuntu 9.04 release.
Amount of time taken to shutdown, from button being clicked to machine powering off. Measured in seconds; less is better.
Windows lags a little behind the Linuxes, with 64-bit again proving a sticking point - this time for Windows Vista.
IO testing
To test filesystem performance, we ran four tests: copying large
files from USB to HD, copying large files from HD to HD, copying small
files from USB to HD, and copying small files from HD to HD. The HD to
HD tests copied data from one part of the disk to another as opposed to
copying to a different disk. For reference, the large file test
comprised 39 files in 1 folder, making 399MB in total; the small file
test comprised 2,154 files in 127 folders, making 603MB in total. Each
of these tests were done with write caching disabled to ensure the full
write had taken place.
Amount of time taken to copy the small files from a USB flash drive to hard disk. Measured in seconds; less is better.
Amount of time taken to copy the small files from one place to
another on a single hard disk. Measured in seconds; less is better.
Let us take this opportunity to remind readers that Windows 7 is still at least nine months from release.
Amount of time taken to copy the large files from a USB flash drive to hard disk. Measured in seconds; less is better.
Amount of time taken to copy the large files from one place to
another on a single hard disk. Measured in seconds; less is better.
With the exception of Windows 7 while copying larges files around a
hard drive, Windows generally suffered compared to Linux in all of
these tests. Obviously Windows does have to worry about some things
that Linux doesn't, namely DRM checks, but these figures show a drastic
performance difference between the two.
Notes: Vista and Windows 7 really seemed to struggle with copying
lots of small files, but clearly it's something more than a dodgy
driver because some of the large-file speeds are incredible in Windows
7.
Both Vista and Windows 7 seemed to introduce random delays when
deleting files. For example, about one in three times when deleting the
files from our filesystem benchmark, this screen below would appear and
do nothing for 25-30 seconds before suddenly springing into action and
deleting the files. However, this wasn't part of our benchmark, so
isn't included in the numbers above.
This was very annoying.
Richards benchmark
Notes: This was done using the cross-platform Python port of
Richards. For reference, Ubuntu 8.10 uses Python 2.5.2, Ubuntu 9.04
uses Python 2.5.4, and we used Python 2.5.4 on the Windows tests. Even
though the 64-bit results for Linux and Windows don't look that far
apart, we have to admit to being very impressed with the Windows tests
- the deviation between tests was just 3ms on Vista, and 5ms on Windows
7, compared to 20ms on Linux.
Amount of time taken to execute the Python Richards benchmark. Measured in milliseconds; less is better.
It's clear from that graph that having a 64-bit OS can make a real
difference in compute-intensive tasks, but it's not too pleasing to see
Windows pip Linux to the post in nearly all results.
Switching to ext4
All the Linux benchmarks above were done using ext3, so what happens when we switch to ext4? Well, not a lot:
Boot, shutdown and filesystem tests for Ubuntu 9.04/x86-64 using
ext3 (blue) and ext4 (red). Measured in seconds; less is better.
Although there's no difference in shutdown speed, the boot time
using ext4 dropped by 8 seconds, which is a fair improvement. We can
probably discount the the USB to HD tests simply out of error margin,
which leaves the HD to HD tests, and there we find a very healthy
boost: 3.7 seconds were shaved off the small files test, making ext4
about 25% faster. Our tests also showed an improvement in the large
file test, but it's not as marked.
Conclusions
Benchmarks are always plagued with questions, uncertainties, error
margins and other complexities, which is why we're not going to try to
look too deeply into these figures. Obviously we're Linux users
ourselves, but our tests have shown that there are some places where
Windows 7 really is making some improvement and that's good for
competition in the long term. However, Linux isn't sitting still: with
ext4 now stable we expect it to be adopted into distros fairly quickly.
Sadly it looks like Ubuntu 9.04 won't be among the first distros to
make the switch, so users looking to get the best performance from
their Linux boxes will either have to fiddle with the default options,
have patience, or jump ship to Fedora - which will be switching to ext4 in the next release..
www.tuxradar.com
