CS488 Project Log

Jeremy Sharpe (ja3sharp, 20250138) - “Saxophone Hero”

Wednesday Nov 17, 3:34 PM

I am starting a bit later than I would have liked, but I’ve finally begun work on my final project for CS 488.

The initial plan is to take my Assignment 3 and enhance it to be drawn in a cel shaded style as a proof of concept.

4:43 PM

Successfully converted from GLUT spheres to spheres drawn with triangle strips. This is important so that I have control over every drawn vertex (and hence on the texture coordinates used for each vertex), which apparently the GLUT sphere does not really provide.

Next, going to attempt to apply black borders.

8:39 PM

After much gnashing of teeth, cel shading has been achieved on the A3 puppet:

A lot of stuff is unfortunately done in software (all lighting calculations are done with Matrix4x4s and Vector3Ds, for instance) and it’s kind of slow. Need to figure out how to effectively offload work to display lists. Obviously the exact parameters of the shading could be tweaked as well. Also the nose is surrounded by this weird artifact.

It may be a little weird to get this to interact nicely with shadows.

Thursday November 18, 2:12 PM

I’ve been playing with tricks to make rendering faster without having to resort to vertex shaders. I would try to use those, but only a small subset of the lab computers support them, and I’d like to avoid using them if possible.

The first obvious trick was to use a display list for the outline drawing - the fact that I wasn’t already doing this shows why it’s a good idea to sleep on things. It's worth noting now that when drawing for the purpose of shadow mapping, we can also use display lists since shading is unnecessary. (As I actually implement shadow mapping, this is an area that may turn out to be more complicated.)

The second trick is to notice that with a fixed light source, we don’t really need to rerender the lighting all the time, and in fact we only need to rerender the lighting for the primitives that move. This means that for anything not moving in the scene, we can use display lists to cache their lighting calculations. Each primitive is assigned its own unique display list to achieve this. (Note a potential tricky interaction with reflections however, as this effectively adds a second light source.)

Taken together, these mean that in my final scene I won’t be doing too many dynamic lighting calculations and staying in software for the cel shading computations should be feasible. Phew.

(Bonus performance tip: the “-O3” compilation flag improves performance by about 3x. I’ll keep that one in my back pocket.)

Friday November 19, 7:23 PM

I’ve spent quite a bit of time trying to get lighting right and make outlines appear consistently. Ported meshes from the ray tracer into this project (with the eventual goal of modelling a saxophone and various other objects), but outlines are working poorly.

9:30 PM

Finished a cool Python script that allows me to generate arbitrary bendy pipes with varying width (for eventual use for the saxophone). As a side effect it allowed me to generate this sweet torus:

Unfortunately I don’t have a good solution (yet) for the horrible aliasing that the cel shading is experiencing. Here’s another badass creation:

Now to actually model something useful with this thing...

10:51 PM

Going for a snack, but protosax is achieved!

This uses Bézier curves to achieve the approximate shape of a sax, and uses my curvy pipe generator to draw it.

Saturday November 20, 2:55 PM

Getting closer (with 6 cubic Bézier sections):

...and closer:

So far this just linearly increases the radius of the sax as it goes from the mouthpiece to the end. The next step is to flare the end and make the middle a bit thicker, without altering the radius of the mouthpiece area.

4:03 PM

Got sidetracked into fixing my mesh aliasing problem, which turned out to be caused by the fact that I wasn’t average face normals for each vertex. Instead, every time I drew the vertex (a total of 4 times, usually) I would use a different face normal, which caused discontinuity in shading for obvious reasons. Unfortunately this adds a second or so to the startup time (as it computes each vertex normal - this would be horrific to do at every render), but it’s not too bad.

The results are gorgeous:

4:45 PM

And the final (for now) saxophone, with flaring:

And I side-benefit of the aliasing work I did. Even though I now have to do (roughly) 4 times the work per vertex (to transform the vertex normal into world coordinates using the current matrix), I can drastically reduce the poly count of my models without them looking terrible, producing a net win for performance.

Monday November 22, 6:41 PM

Did a bunch of work to prettify my musician, and add the saxophone to the puppet model:

I also fixed picking, which I somehow got away with being incredibly broken on assignment 3 (without realizing it - at first glance it seemed fine). It’s necessary to figure out animations and how to make him correctly hold the saxophone.

Tuesday November 23, 9:49 PM

Wow, shadow mapping is a pain in the ass. It’s difficult to debug the transformation from light coordinates to camera coordinates, it suffers from extensive precision errors without careful parameter caressing, and even once it is set up fairly nicely, it is still disgustingly aliased for all but the narrowest light sources. Took around two days to implement to a reasonable state, a lot of which was spent frustrated and trying to figure out debugging techniques.

Although my implementation is based on previous work by others, I had to add make the enhancement of using multiple textures to support cel shading and shadows at the same time.

Here is a simple scene with no shadows:

Here is the same scene with shadow mapping and the light having a wide field of view (but not ridiculously so):

Notice how the shadow of the saxophone plays across the rounded figure of the player (on the chest, leg and arm). Notice also how terrible the aliasing and artifacting is everywhere, especially on the chest and ground.

This is a visualization of the depth map (from the point of view of the light) used to achieve the above effect:

You can see that the player covers a very small section of the depth map, hence why the aliasing is bad. (Notice also that the orientation of this texture is of no consequence, since it’s compensated by the light’s viewing matrix.)

Here is the scene with a significantly narrower (and kind of impractical for actual use) field of view:

Notice the artifaces and aliasing are significantly reduced. The light’s depth map, and the relative size of the player in it, shows why this is true:

A trivial solution to this aliasing problem would of course to use multiple, small depth maps from the light source. Unfortunately, this is not computationally feasible in my model. I’m already pushing it adding shadows at all, given that my cel shading is done in software.

Wednesday November 24, 4:26 PM

I’m working on efficiency today since I’m getting kind of worried about my 50 - 70 ms render times with shadows enabled and still very simple models. I was having trouble a while ago getting cel shading to work inside display lists, but I finally figured out that it was because I was requesting the MODELVIEW matrix (with which to do manual transformations) after performing a rotation inside the display list. Since none of the transformations are applied to the actual matrices when compiling display lists, this resulted in the wrong matrix being retrieved.

Preliminary results show that display list use for my spheres drops render time by 20 ms with shadows on. Because of the nature of my lighting, I’ll have to recalculate the display lists when spheres are rotated or moved, but this should greatly improve my render times assuming most objects are static.

5:05 PM

That was a surprisingly quick refactor, and actually made the code for both the spheres and the meshes much simpler. When not moving a ridiculous number of things at once, render time is now significantly faster.

8:29 PM

I’ve been reading up on vertex shaders, and decided that it wouldn’t be too difficult to create a vertex shader to replicate the work I’m currently doing in software to compute the 1D texture coordinates of the lighting.

In the course of trying this, I had to log on to one of the machines that supports shaders. Wow! This machine is so much faster than the other I was using. I’m literally getting < 0, 1 and 2 ms render times for most renders, and > 70 fps even for frames with heavy in-software lighting calculations. I think I’ll reprioritize the shader work behind some of the more directly objective-based work since I don’t think I need the speed as much as I thought I did.

9:36 PM

Just spent half an hour solving a really stupid bug that ended up being caused by me binding a texture 10 lines too late. Sigh.

Thursday November 25, 3:15 PM

Just got texture mapping working. Made a lot of little stupid mistakes that made this take longer than it should have, mainly because I am now dealing with three textures simultaneously (one for shadows, one for lighting, and now one actual texture for texture mapping), and keeping them all straight is a bit complicated. If I had realized how much texturing there would be from the start, I could probably have kept things much more organized.

For example, for multitexturing you have to keep switching between the various different texture modes, and it would have made my life much easier if I had required that every block of code dealing with a particular texture mode needed to explicitly switch into that mode (with a single texture, you can just assume you’re always in the right mode, so it seemed okay not to be explicit about this at first).

In any case, the result is cool:

(This is just some crappy texture I drew for demonstration purposes.)

Friday November 26, 8:53 PM

Sweet. I got sound working (with SDL_Mixer, which I (and the internet) highly, highly recommend over OpenAL, especially since the sample code provided for it actually almost compiles, unlike the OpenAL code - protip, try changing audio_rate from 22050 to 44100 if you get audio artifacts).

I now have the intro to Giant Steps by John Coltrane playing in the background. I’m going to get so sick of it soon, but for now it’s just awesome.

Now to make the game playable.

Saturday November 27, 4:53 PM

I’ve been working on game logic and the game’s note board. Got something pretty cool I think:

It even scrolls as time goes by!

Now I can’t decide if this note board is horrible programmer art, or actually in the style of the game.

6:24 PM

This is how I’m supplying the notes to the game. I opened up Goldwave and looked at the waveform to find where each note occurs to millisecond level, and then put that in a note chart file:

It was tedious, but if I did it well enough, I should be done with charting forever. (In this picture I’ve only assigned the notes for the first few.)

9:17 PM

I now have a board with notes, although they are not yet playable. They are surprisingly fairly well synced with the music, although they do drift out of alignment slowly (I may be able to combat this by restarting the music once in a while).

10:24 PM

Did some work with the sync and got it to resync itself every loop through the song, which is nice. I also figured out why I wasn’t getting outlines, and put alpha on the notes as they come in at the back. It’s much prettier now:

Now the tricky part: interactivity.

Friday December 3, 3:53 PM

I’ve been furiously working away on this and other assignments.

The game is now fully playable. Sound turns off when notes are missed and back on when notes are hit, there is a miss sound effect, notes explode when hit, and notes float out of the end of the sax:

And now I’m working on the last purely graphical component: reflections. After that comes animations (both of characters and the camera).

5:51 PM

I have fully functional reflections! Albeit without use of the stencil buffer, since it is unnecessary for my purpose.

I even have tricky behaviour so that the sprite particles show up properly in the reflection, which was non-trivial.

Now to build a wall to contain the window.

10:38 PM

I’ve added walls (and changed the window colour from its previous, hideous green):

Saturday December 4, 1:40 PM

I’ve now attained camera animations using quaternion interpolation (I have code that translates my rotation matrix to a quaternion and back). It’s still hard-coded in, so I need to make a way for the camera animations to be scripted from the Lua specification. To aid myself in actually developing the script for the camera, I’ve added a command that prints the current rotation as a quaternion and the current translation.

I also still need to add easing of these animations so they don’t look jerky. It might even be a good idea to use Bezier curves to model the camera’s animation so it follows smooth arcs, but this might take a while, so I'll hold off. May want to just special case things like circular motion.

2:08 PM

Well that was easy. I tried to find something on the internet to explain the basics of non-linear interpolation (for camera easing), but couldn’t find anything. So I just made it up myself and about the second thing I tried was exactly perfect (for my purposes anyway). In hindsight it’s kind of obvious, given that it’s called non-linear interpolation.

4:24 PM

Scripted camera animations now work. They are specified in a file like this:

where the first line is the length of the script in milliseconds, and each subsequent line represents a smooth motion of the camera between two different rotations (expressed in quaternions) and two different positions. This could be extended a bit to allow for selection of start and end easing functions for each motion (this would allow for nice circular motion, for example).

6:38 PM

Just completed the addition of translational animations, specified in the Lua file. They were straightforward if tedious to add, due to a new parameter of time needing to be pushed through all walk_gl and shoot_particle calls (since the position of models is now a function of time). I had an issue where I forgot to all translate the origins of particle emitters when particles were created, but this was easy to fix.

As a proof of concept, my character is now levitating up and down constantly.

And now, the part that really matter (it’s possible I won’t even use translation animations in my final product unless I attempt walking): joint animations. Hopefully not too much more difficult, although I may run into problems since I’m going to be interpolating Euler rotations, which can get nasty.

7:53 PM

Completed implementing joint rotation animations. It’s not difficult to do, the only question will be if the Euler rotations are expressive (and predictable) enough to accomplish the animations I want. Most of my animations I anticipate being one dimensional joint rotations, so I should be more or less fine. So far I have my model raising and lowering his arms, and raising and lowering his head.

I mentioned somewhere earlier that recalculating the cel shaded lighting was expensive, and that it would have to be done whenever objects rotated or moved in the scene. It turns out that I can choose to only recalculate the lighting every 20 or so frames, and it still looks fine. This is a huge increase in speed.

Sunday December 5, 2:56 PM

I’m making a piano:

4:56 PM

The band begins to emerge:

Question is, do I have time now to add a drum and/or a bass? The first is probably not too bad with a bunch of cylinders (although orienting them in a believable way seems tricky). The second would probably require a fairly complicated script to generate, so I may skip it.

Monday December 6, 3:06 PM

I’ve been working feverishly since about 10 this morning (after leaving the lab last night at 11!), breaking to go to class and have lunch. There is one new character in the scene, the one person who isn’t allowed to abandon the concert because it’s a 25 second perpetual loop - the bartender:

Soon he’ll have smoke coming from his cigarette.

The club has become much more complete with the addition of furniture of various types (and a texture mapped liquor cabinet with my beautifully drawn texture):

The floor has also been changed to look less like sand.

The end is in sight (and thank god, because look at those frame rates!). Now the final touches - cigarette smoke and animations that respond to how well the player is doing (by going slower as the player gets worse). Although now that I think about it, I want to redo the camera script so that it is less stomach churning - also I forgot quaternion normalization in my implementation. Augh details!

4:16 PM

Smoke is now happening. Cel shaded circular smoke to be precise. The bartender bops his head around with the music, making the smoke take on a nice pattern:

5:03 PM

Fixed normalization of quaternions. Hopefully this should fix the egregious perspective distortion that occurs on some extreme camera motions.

The final technological hurdle: animations that respond to how well the player is doing.

10:03 PM

The player and the bartender’s animation now grow slower as the player misses notes, and stop if the player’s “score” (a number than ranges from 0 to 20) reaches 0. Start playing again, and the scene comes back to life.

As one of my final acts, I finally stamped out a bug that seemingly occurred randomly, and manifested itself as an OpenGL error 0x502 (invalid operation). Forgot to initialize an int representing a texture to 0. Oops! Moral of the story - check for OpenGL errors (not that they make it easy - in the end, instead of manually checking for errors after every operation, I just sprinkled checks semi-randomly throughout my code, and tracked down problems more precisely as they occurred).

And with that, I’m done! The game is now in a presentable form. Obviously some tweaks could still be made, but as of right now I’ve basically fulfilled the vision that I set out to create 3 weeks ago.

Now to practice the demo tomorrow and do the real demo on Wednesday. (Also still need to write documentation!)

Tuesday, December 7, 3:56 PM

Spent the day writing documentation (and to my shame doing some last minute code reorganization and fixes) and handed in the project. Demo tomorrow.

Thursday, December 9

I’ve uploaded a couple of videos of the completed project:

Short gameplay demonstration with sound: http://www.youtube.com/watch?v=Hjp3t6JTIAs

Smooth, but with no sound: http://www.youtube.com/watch?v=qC5WhZ96OI8

Choppy, but with sound: http://www.youtube.com/watch?v=6SqFjKlxQyw