The Windows Cache Manager - A Quick Overview

Well, if I have to spend my days (and currently nights!) with Windows cache and memory management, it should at least be good for a blog post ;)

The Windows Cache Manager is a central operating system facility managing the system cache, which transparently supports all Windows filesystems. It is capable of handling file data - including multiple data streams for filesystems that support such - as well as filesystem metadata. Since the operating system provides for a centralized external storage caching mechanism (including support for network and disk based access), filesystems are alleviated of implementing proprietary caching functionality.

The Cache Manager operates on a relatively high level for a kernel mode component, interpreting files as byte streams vs. individual disk blocks on physical storage media. Abstracting away from physical file layout allows for simplified analysis of data access patterns and resulting optimizations for instance in terms of better read-ahead performance.

A segment of the kernel virtual address space is assigned to the system cache by the Windows Memory Manager. The Cache Manager further slices this segment into 256KB chunks to hold cached views of filesystem data.

Mapping file contents from disk into the system cache is 256KB-aligned - that is, when reading 100 bytes from a 1MB file at byte offset 400,000 the Cache Manager will fill up the corresponding 256KB chunk in the system cache starting at byte offset 262144 through 524288-1. Assignment of filesystem mappings into the system cache 256KB slots happens in a round robin fashion.

The filesystem tracks the physical bits of a logical file and the type of I/O requested (buffered vs. unbuffered) within corresponding file objects. Multiple file objects can represent the same file on disk. The former share a "shared cache map" which serves as metadata structure for the file on disk being represented by the file objects. It contains a mapping from file offset for a particular view to system cache virtual address space. The Cache Manager utilizes the shared cache map to ensure cache coherency among the different file objects for the same file - for instance by only caching the same view into the file exactly once. Data relevant to individual file objects alone - that is, access patterns to handle read-ahead - is kept in object-specific private cache maps.

The Win32 API exposes flags to influence cache read-ahead behavior via FILE_FLAG_SEQUENTIAL_SCAN and FILE_FLAG_RANDOM_ACCESS When not specifying any of the file flags at all, the Cache Manager tries to analyze and determine an access pattern (strident access, sequential forward or backward scan) in order to optimize its performance.

A number of lazy-writing threads that wake up periodically, sequentially go through the system cache and flush out a predetermined number of modified 256KB chunks to persistent storage in each wake-period. A list of linked shared cache maps indicates system cache chunks that have been written to and thus need to be considered by the lazy-writing threads.

It is interesting to point out, that with Windows' particular kind of memory mapped file caching, the Cache Manager remains oblivious to which parts of the cache actually reside in physical memory (RAM). Instead, the Windows Memory Manager is free to optimize physical memory utilization to balance system cache vs. demands of user processes.

The overall physical size of the system cache depends on the operating system - for 32Bit systems it maxes out at a little over 500MB. As for the size of the kernel virtual address space assigned to the system cache, it depends on the amount of RAM in the system and is calculated via the formula: 128 + (RAM in MB - 16) / 4 * 64.

Resources:

• Fourth Edition of Russinovich's  Windows® Internals, Fourth Edition - an updated version has been released in June 2009, covering the more recent versions of the OS: Windows Internals, Fifth Edition.
• 2005 MSDN Channel 9 interview with Molly Brown, the engineer responsible for the Vista version of the Windows Cache Manager: part 1 and part 2

Interview in Germany's #1 College Magazine: The Google Experience and CS Careers

In November last year, Dominik Hahn, editor at presstige and fan of a couple of the podcasts that I'm collaborating on, contacted me for an interview in their publication. Little did I know at that time, that their magazine boasts a print run of 10.000 issues. IMHO, that is pretty impressive for an entirely student run magazine.

He was particularly interested in career advice I could give for fellow CS students and my personal experiences with working for Google.

I have embedded the magazine, neatly turned to the right page for your viewing/reading pleasure, right after the jump.

If you don't speak German, let me distill the article for you: Yes, go work for Google.

It's a special place, where you are given any and every chance to evolve as an individual and engineer. This is not to say that there aren't any other workplaces that qualify, but Google definitely gets a lot of the things I was looking for right. And then some.

Open publication - Free publishing - More presstige

WoW: Wrath of the Lich King Review for games™

Oh Noes! A fan of our podcasts just dropped me an email, inquiring about WoW matters and made me realize that I totally forgot to plug my very own review of World of Warcraft: Wrath of the Lich King in games™ issue 01/09.

When Soenke Siemens, associate editor in chief of German games™, asked me for a WotLK review for their print publication in October 2008, how was I supposed to say "No!". Especially given that such an inquiry translated to an excuse for burning time hunting down Arthas in Blizzard's latest extension to Warcraft lore?

My personal history with WoW started as early as fall 2004 with the US Beta and has been unfolding ever since. Admittedly, the journey hasn't been without frequent pauses of several months. After all, real life obligations require their share of attention and levelling up too :)

So, I guess it shouldn't be surprising that after my initial three weeks of intensive adventuring in Northrend, thanks to a beta key provided by good friend and former Google colleague Sumer O. (Blizzard employee, now) and the big ol' collector's edition box by the magazine, I had to attend to my other - i.e. RL - character, Mustafa.

I might get around to pick up a little WoW-ing over the course of the next four weeks though. If you feel like teaming up with our friends & family guild on German server Khaz'goroth, I suggest you drop me a note. I have a dormant US account too, but chances for me to pick up that one too, while living in Germany and barely finding time to dwell in the European server pool, are currently very low.

Distilling the essence of the massive expansion to a two page review wasn't easy, especially when aiming for an original article that does not mimic a fact sheet or worse resembles little more than a mutated press release.

Judging from the feedback we received for the article, coming up with a compelling review seems to have worked though. Thanks to everybody for the kind words and feedback.

If you're interested and lucky, you might still be able to pick up a copy of the issue at a newsstand of your choice, before the February wave of magazines washes it all away.

On the other hand, thanks to games™'s PDF offer, you can always opt for the cellulose-free way of digital distribution and get access to any back issue you'd like ;)

Review: I Love Katamari (iPhone/iPod Touch)

Roughly a week ago, in a surprise move, Namco released an iPhone game based on one of their more recent and creative IPs, that is Katamari Damacy. The official title of the version for Apple's handheld combo is dubbed I Love Katamari.

The basic game mechanic remains unchanged to what fans of the series have come to expect from the various incarnations of the title for the PSP, PlayStation 2 and Xbox 360.

The goal is to roll a sticky ball, that is a Katamari in the game's lingo, around and snowball it to a certain size by rolling over items strewn around in 3D stages.

As you roll around the first level of five for the iPhone title, you quickly realize the nature of the challenge ahead: you can't just roll up objects in any arbitrary order.

Items of a certain size will only be picked up by a Katamari that has reached a respective minimum volume. Thus you are required to iteratively identify objects you can pick up as the monstrosity you are pushing through the level reaches bigger and bigger sizes.

I Love Katamari features four modes which provide for slight variations while keeping the core game mechanic intact. In Story Mode you get a taste of the crazy background of the series, where you are prince to the King of All Cosmos, who requests you to roll up a particular object, varying from stage to stage and involving items such as a dog or a truck.

You start out with a blank Katamari that is just big enough to pick up the smallest items, depending on the stage setting these range from sushi to footballs.

A tight time limit forces you to hurry up and grow your Katamari as fast as possible in order to be able to pick up the requested object. Stages for all playmodes are made available subsequently through successfully completing challenges in story mode.

Once you have cleared a stage it is available in any of the other three modes. Time Attack gives you the opportunity to roll up a Katamari as big as possible within a set time limit, Exact Size Challenge requires you to do exactly that and Eternal Mode is your chance to enjoy stages without having to watch any limitations while building a behemoth to your liking.

The standout feature of the iPhone/iPod Touch version of the game are the controls. Navigation through the leves is accomplished by tilting and turning the device in a very intuitive fashion. To me this feature made me grok the Katamari Damacy fascination for the first time, despite having played the PS2 version for a short while.

From my personal experience I consider the game absolutely suitable for short bursts of playtime. Over the last five days, I have fired up the title every single day in order to try and please the King of all Cosmos while riding the subway or waiting in line.

Unfortunately not all that shines is gold and thanks to frequent slowdowns, occasional sluggishness and control issues every now and then, I Love Katamari makes sure we don't forget.

More than the obvious technical bugs, I am disappointed by sloppy design decisions. For instance, at certain points your Katamari goes through a distinct growth phase - which by the way always causes a slowdown and makes the framerate drop to something around 2-3 fps - where I'd expect to see some kind of consequence to the sudden growth. 

Something along the lines of all of a sudden being able to pick up significantly larger objects than was possible before. That usually doesn't happen though until you've rolled over a couple of more items, that would have already stuck to the Katamari before the growth sequence, which doesn't help to make sense of the whole growth animation/framerate meltdown.

Despite the issues that the game suffers from and which Namco will hopefully address in a future update, my verdict is very positive. This is quite in contrast to the conclusion Luke Plunkett draws in his micro-review on Kotaku and Chris Kohler on his Game|Life blog on Wired.

If you are in for a game that does not mimic the bad half of 80s game concepts - as they seem to be flooding the AppStore - I suggest you grab a copy of I Love Katamari and start nagging Namco to release a patch ASAP - not telling them of course that you already enjoy the game - as I do ;)

Update - Dec. 22nd, 2008: Namco has released an updated version of the game which resolves most of the technical issues I Love Katamari was suffering from. The title in version 1.0.1 plays much smoother now.
The bug fixes also result in a significantly decreased difficulty level. For instance I was trying to get past the Park stage for the last two days, but failed miserably each and every time I attempted to. After the patch, it took me exactly one try to roll up the requested light truck and pass the level.

The UI stayed pretty much unchanged, aside from an element showing the extent of tilt the iPhone/iPod Touch sensor reports to the game. I suggest you either update your version of the game, if you have purchased it already that is, or use the new release to get it in the first place ;)

Space Siege Preview for games™

My Space Siege Preview for gamesTM, originally uploaded by Mustafa K. Isik.

As listeners on Germany's most popular gaming podcast games und so already know, Alex and I met up with Chris Taylor and talked about his studio's latest game Space Siege.

gamesTM asked me for a preview of the title which has been published in their latest, i.e. August 2008, issue. The magazine is available at newsstands and bookstores as well as a slick DRM-free PDF online.

It is jam-packed with previews (e.g. Civ IV: Colonization, CoD: World at War, Treyarch's Bond, Dawn of War 2), reviews (Battlefield Bad Company, Alone in the Dark) and a LucasArts Indy retro flashback - well worth shelling out the bucks ;)

Google Tech Talks: Wiring Hacker Synapses & More

Some of the talks given at EclipseDay at the Googleplex are available as videos via the Google engEDU Tech Talks Channel on YouTube now.

For your (and my) convenience, I have embedded our very own Eclipse Communication Framework Wiring Hacker Synapses talk right after the jump. But if you are really in for a ride to remember, I suggest you indulge yourself in the full list of EclipseDay Grand Teton room recordings.

• Eclipse @ eBay, Michael Galpin
• How Mylyn Changes the Way I Develop, Bjorn Freeman-Benson
• GWT in Eclipse, Bruce Johnson
• Plug-in Development Tips and Tricks, Chris Aniszcyk
• and Wiring Hacker Synapses, Mustafa K. Isik & Scott Lewis

Wait - there is even more ;) Photos.

Visiting the Friendfeed Global International Headquarters ...

... and helping out with the move.

Since I was in Mountain View, CA for the first half of this passing week, I used the chance to make up for all the missed friendfeed open house and TGIFF invitations. On Monday afternoon I took a break from preparing my slides for EclipseDay at the Googleplex and dropped by the Friendfeed Global International Headquarters - new and old.
It was great to finally meet up with the team behind the service of which not only I think that it is excellent in scope, execution and promise.
After a warm welcome by Ana and a chat with Paul, I joined the team in moving from the old office to the new location, contributing my share to friendfeed's continued success ;)

Among the non-trivial tasks of relocating a global international headquarter was mounting a high-tech mailbox to the precise specifications laid out by Ana - that is "So that our short and chubby mailman can reach it". Paul and I had to make more than one attempt to get it right.

Over the course of my stay I was successively introduced to the rest of the team and I have to say you guys rock. I had a blast talking to you all and I am sure that my Monday afternoon friendfeed diversion helped with the success of the talk at Google the following day :D

P.S.: Ana thank you for the shirts. I took mine out for a walk in Munich today. My girlfriend says thank you for thinking of one for her too - since you and I could not find the girl cuts in the box, it looks like as if I'll get around to wear that one too though :)

Corresponding items on my friendfeed:

• blog post
• T-Shirt flickr photo

Hot Off the Press: Slides for Cola Tech Talk "Wiring Hacker Synapses"

These are the slides for my Cola: Real-Time Shared Editing talk at Google for EclipseDay.

You can download the PDF version of the slides by clicking through to the scribd page for this presentation.

The embedded slides/PDF version have been stripped off any animation, that is are fully built and thus should be well suited for non-presentation style consumption.

Wiring Hacker Synapses - Upload a Document to Scribd

Read this document on Scribd: Wiring Hacker Synapses

Cola: Real-Time Shared Editing - Screencast

The Ganymede release is looming on the horizon and the Eclipse community is busy wrapping things up. Our team - the Eclipse Communication Framework - is no exception there.

In the course of getting ready for launch, I have put together a short screencast, showcasing the first incarnation of "Cola: Real-Time Shared Editing" for Eclipse.

Follow my friend Thomas K. and me, lead you through an exemplary shared editing session.


Cola: Real-Time Shared Editing from Mustafa K. Isik on Vimeo.

You can enjoy high resolution versions of the screencast at the vimeo page (higher than the embedded version) or by clicking on the HD icon when doing a mouseover (full 720p glory).

EclipseDay hosted by Google: Attendee list filling up fast

On Tuesday June 24th, 2008 we will hold an Eclipse event at the Googleplex. The first of its kind, hosted by the great folks of the Google Open Source Programs Office (OSPO).

As Ian noted on his blog, I suggested an event of this type while I was interning with the Build Tools team at Google last year. Soliciting the eclipse committers mailing list for feedback concerning a potential DemoCamp at Google revealed great interest in the community and knowing of Google Engineers' love for all things technologically slick - and Eclipse definitely qualifies ;) - ultimately led to an event of larger scope.

Ian Skerrett, Marketing Director for the Eclipse Foundation, Leslie Hawthorn, Program Manager for Google's OSPO and Rob Peterson, Manager of the Google Build Tools team, signed up for the idea right away.

We started brainstorming ideas for the conference and bouncing them off of each other via emails and phone conferences late last year and have since been supported by more members of the Google Build Tools team and OSPO. I would like to single out my host Rob Konigsberg, colleague Rob Clevenger and Tiffany Griffith for their efforts on making this event happen.

Even with so many people involved and the event having developed a dynamic of its own, I like to think of EclipseDay as my personal Waynestock and if you attend, it shall be yours too.
I hope for a similarly fun get-together :)

The Google-hosted very first EclipseDay will manifest as a gathering of one hundred eclipse-enthusiast developers. The talks will provide for a general hands-on approach and should prove to be insightful for a broad audience. I suggest you have a look at the agenda.

Space is limited so please sign up if you would like to join - only nine spots are left on the attendee list.

It would be great if you could make it to the talk Scott Lewis and I will give on ECF work: "Wiring Hacker Synapses: Collaborative Coding and Team Tooling in Eclipse"

See you at the Googleplex.


View Larger Map