As I’m going through testing and putting the finishing touches to my next release, I thought I’d let you all have a peek into adding a seemingly simple feature. I’m about to release a new version with a slideshow feature added, so you can just assign an album as a source, and my app will just go through the photos. I’m just doing some testing manually before submitting to the app store.
It turned out to be an interesting problem. The code to do it really isn’t all that complex, load the photo as a texture, and do matrix manipulations to achieve the transitions, or use a shader. I ran into some troubles using the GLKTextureLoader, but they turned out to be only related to the simulator. I started using the GLK tools more as it provided an easy way to load texture asynchronously, and while I already had openGL code to load images to texture, I didn’t fancy the extra work of writing all the stuff to make it load textures in a work thread. Ultimately, if I ever want to convert this to a multi platform app, then the less I rely on Apple specific frameworks, the better, however, pre written frameworks are a massive time saver.
The biggest problem was on how to present a slideshow maker to the user in my context. Having a predetermined video format of 16:9 or 4:3 would make it a lot easier, but with my app, i want my users to have total freedom, but I don’t want to present a complex interface. The app has to cope with formats that apply to any presentation. Beginners need to have something to walk them through a simple setup, but more advanced users need to be able to tweak the settings. A modern slideshow consists of portraits and landscapes, mostly in 4:3, but it’s so easy to crop a picture now, it’s not a reliable standard, nor is it one I want to encourage. With projection mapping, any surface can be a presentation surface, and not likely to adhere to a 4:3 or 16:9 aspect ratio. Eventually I decided to adopt a broad approach, anticipating my users to supply images in a singular aspect ratio, my app will broadly guess, landscape, portrait or square, or the majority of anything in between.
And then, once you accept that while there are ways to work around aspect ratios, which my app offers, letterboxing is another feature you have to write in. I though I might be able to do it with some extra drawing of black boxes, that concept does not work with something like a fade transition. Your textures must go through an addtional step to letterbox them before going through to a shader, easily achieved through a few lines of code, but a processor and memory intensive operation. here’s the code:
- (UIImage *)addLetterboxToImage:(UIImage *) inputImage
{
CGFloat inputWidth = inputImage.size.width;
CGFloat inputHeight = inputImage.size.height;
CGFloat fitWidth = self.textureBounds.size.width;
CGFloat fitHeight = self.textureBounds.size.height;
CGFloat inputAspect = inputWidth / inputHeight;
CGFloat fitAspect = fitWidth / fitHeight;
UIImage *letterboxedImage;
if (inputAspect == fitAspect) {
// No need to do anything
letterboxedImage = inputImage;
} else {
CGFloat scale = inputAspect >= fitAspect ? fitWidth / inputWidth : fitHeight / inputHeight;
UIGraphicsBeginImageContext(self.textureBounds.size);
CGFloat newWidth = inputWidth * scale;
CGFloat newHeight = inputHeight * scale;
CGFloat newOriginx = (fitWidth / 2.0) - (newWidth / 2.0);
CGFloat newOriginy = (fitHeight / 2.0) - (newHeight / 2.0);
[inputImage drawInRect:CGRectMake(newOriginx, newOriginy, newWidth, newHeight)];
letterboxedImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
}
return letterboxedImage;
}
In short, most of my work for this feature went into UI work, the actual code to make it was relatively simple. Here are the logs:
40 minutes ago: Sliders now change dimensions. I think that's about it, few more cosmetic changes but all the functionality is there. 63 minutes ago: Change maximum size for the slider to that of the largest dimension of the config, same as we use to calculate. 64 minutes ago: Code clean 80 minutes ago: Basing dimension size, using the largest dimension of the config. 2 hours ago: Save slideshow on return to editor fields. 3 hours ago: Slide show fields editor now updates after slider views. Removed log messages. Still a problem with the largest and smallest dimensions selector. If a large one is scaled down, it could end up being smaller than the smallest image. 4 hours ago: Code tidy 4 hours ago: stop loading texture twice. 4 hours ago: Don't forget to delete the new textures. 4 hours ago: Removed the now unused ivars. Current solution to just recreate the image with correct dimensions means we won't run into any problems with fades. The generalised methods for VAOs, coords and colours are still useful, so we'll keep them in. 5 hours ago: Letterboxing done asynchronously so the whole next texture loading cylce is off the main thread. 5 hours ago: Added method to do the letterboxing, it's being called but I've not got the dimensions right yet and it's making them too large. 17 hours ago: Added ivars for vert coords of current and next texture. 17 hours ago: Added new ivars for the stuff we're going to need to do letterboxing. 18 hours ago: Use more generalised method to create the VAO for the slideshow. 19 hours ago: Don't forget to pass in the colourBuffer! colour buffer generator now works. 20 hours ago: Move properties to ivars for things we never access outside of class 20 hours ago: Added facility to add second set of texCoords to the VAO. 20 hours ago: Ah ha! you need to keep track of the allocated size, and use that for glBufferData when creating array sizes dynamically. Otherwise sizeof() gives the size of the pointer, apparently.. 24 hours ago: Trying to add a count to the colours is confusing. Didn't know you cant create arrays with dynamic number of elements, malloc is used instead, but I'm getting some very strange results man.... 25 hours ago: Added a count so we can use the general method for larger cases. 25 hours ago: Added more general methods to create vaos, tex coords and colour buffers. 26 hours ago: Added a colourOnly bool to the basic shader, so we can make coloured quads without switching shaders in the slideshow. This way we can add black bars for when we have to do an aspect fit. We should be able to do aspect fitting through matrices or seperate VAOs for each texture. 26 hours ago: Fixed right left transitions. 29 hours ago: Remove debug comments. 2 days ago: Can update dimensions from the pre configured options. 2 days ago: Compare all dimensions by area 2 days ago: Added a convenience property to get the current definition from the app delegate 2 days ago: Can select rows in the dimensions editor 2 days ago: Remove unnecessary observers, analyse dimensions on previous edit storyboard and pass on to the dimensions editing. It's not possible to change the source in this view, just being over zealous. 2 days ago: Dimensions editing locks out of all images in a slideshow are the same format and size. 2 days ago: Removed comment 2 days ago: Set new path, name and dimensions when slideshow is changed. 3 days ago: Creating a new slideshow uses new analyse methods. 3 days ago: Removed keys for recommended format and just store the key with the relevant values, then it provides an easy way to access the correct array of size values. 3 days ago: Improved size analysis, so a preferred format is found, and the sizes are sorted. Also used it to replace information coming from the albums collection view. Cleaned up code. 3 days ago: More annotations added 3 days ago: Added some annotations to properties we can be sure of the type. 3 days ago: Shouldn't loop in set, just use containsObject 3 days ago: Code cleaning and removal of comments. 3 days ago: Observers added and removed as appropriate, stops error on going back from dimensions editor. 3 days ago: Disabling fields now works. Cell index paths couldn't be worked out when it was called, so they've been hard coded. Analysing is better too, but not sure it should be on the source. 4 days ago: Fleshed out the analyse class methods, starting to implement editor fields updates. 7 days ago: Going to move analyse dimesnions method to a class method on the SlideShow, it's too useful and should be used during creation and updating. Provide constant dictionary keys. 8 days ago: More improvement to the dimensions, started to put code in to work out the dimensions. 9 days ago: Added table view for picking dimensions, but not sure if I like it at all. 9 days ago: Avoid division by zero 12 days ago: Added multiplier and unit to speed values in slide show editor. 12 days ago: Support for new multiplier for the speed. 12 days ago: New group for general purpose utility views like the enum and slider views. 12 days ago: MAde slider view more customisable, can add a multiplier and a unit for the label. 12 days ago: Can edit crop style, speed and transition speed. Fixed localized language switch. 2 weeks ago: All current transitions now correct 2 weeks ago: Fader works! Typo in the shader means only the first texture was referenced. 2 weeks ago: Trying to get fader to work, doesn't seem to be picking up second texture, but it's def receiving the mix value 2 weeks ago: Can change transition styles, all slide overs work correctly, but the fade isn't, and the slides with both textures have the underlying textures moving in the wrong direction. 2 weeks ago: Added fade transition 2 weeks ago: Cleaned unused code, added some commenting. 2 weeks ago: Rendered objects could get out of sync with each the external display and editor views. So unified timing to be app wide. 2 weeks ago: Added seperate methods for rendering the transition for clarity. Unified transition styles to switch cases for all up/down/left/right cases. Removed debugs 2 weeks ago: Transition correct up down. 2 weeks ago: Enable fbo, transition now works, just need right transitions. 2 weeks ago: Fix order of operations 2 weeks ago: Slide show advances through the album, but the animations aren't showing yet, probably a mistake in the translation matrix. 2 weeks ago: Moved code to check if transition finished to start of draw method, so texture can be deleted at right time, the next texture swapped in to the current texture, and call the next texture load so it loads asynchronously ASAP. 2 weeks ago: Method renaming 2 weeks ago: Removed ugly cut paste code with nice little loop. 2 weeks ago: Swapped out individual const assignments to a much more sensible enum. 3 weeks ago: Playing with fbo 3 weeks ago: Some commenting and tidying. 3 weeks ago: Set delegate for the glkview, now it's rendering, but the texture has a blue tint, or is only blue. 3 weeks ago: Fix some clear colors. 3 weeks ago: Not unbinding texture and vao was caisomg exc_bad_Access, but still not drawing. 3 weeks ago: Added code to load first texture in the slideshow synchronously to save complex threading we don't need to deal with yet. 3 weeks ago: Pass round the cgimageref rather than the UIImage for more compatibility between iOS and OSX, we need to retain and release the cgimageref, this doesn't seem quite right, I would have thought the property would have retained it, but maybe with thread interoperability it's not quite so simple. 3 weeks ago: Texture loader now loads the texture, we need to keep a handle on the UIImage until texture load is complete. Keeping the cgimagref is not enough, or maybe required a retain, either way it's working now. 3 weeks ago: Stopped crash by keeping image in a property while loading, looks like the image was getting released. It's still a bit messy, but now there's another error reported by the textureloader. 3 weeks ago: Added a glktextureloader and set it up so we can load textures asynchronously, but it's causing a crash for some reason 3 weeks ago: Base draw code put in and mechanism to fetch photos in place, view controller draws result, now we just need to work out how to do a slideshow. 4 weeks ago: Renderer is created and photo album retrieved. There's going to be some fundamental differences on how this works in iOS and in OSX. iOS will work exclusively through the Photos framework, and OSX will be a bit more fast and loose, probably ignoring photos altogether. We'll cross that bridge later, let's just write iOS. 4 weeks ago: Was only storing the localizedName, we should have been storing the localIdentifier as well, easy fix, path stores the localIdentifier, and name provides the human readable localizedName 4 weeks ago: Fixed storyboard segue, slideshow editor now updates all detail text labels with appropriate information. Gonna move to rendering for a bit of change of pace. 4 weeks ago: Added localizable strings for transition and crop styles. 4 weeks ago: Added slideshow editor table view class and bound cells. Configured embedded segue to forward the slideshow 4 weeks ago: Fixed constraints so we get a view split in two with glview and editor fields getting equal space 4 weeks ago: Thumbnail saved before database save, so cell picks it up. If database fails to save, thumbnail is deleted. 4 weeks ago: Slideshows save with thumbnails, but don't seem to retrieve them all the time, and they don't seem to delete the thumbnail. 4 weeks ago: Now we get all the sizes of the assets in the slideshow, but what shall we do with them? calculate an average size? they should probably be hung onto, but collections aren't static, so it wouldn't be a good idea to store them in the database. 4 weeks ago: Can get result of create slideshow action 4 weeks ago: Can activate the album picker for slideshows from the source view 4 weeks ago: Slideshow album picker now only shows non empty collections with at least one image and get a preview too. 4 weeks ago: Most of the view controllers in for the slide show, not sure if I should be making spaghetti with the view controller to pick albums or use multiple copies of it in the story board. I'm probably going to use it for a new custom image picker too. 4 weeks ago: Added photos framework and started on album or group picker. 4 weeks ago: Basic slideshow editor put in, another glkview and container view with embedded table view. 4 weeks ago: Fleshing out even more, but going to switch to updating storyboard so can have data to work with, we'll implement this feature in OSX second as iOS seems to be doing much better. OSX will require a slightly different approach too. 4 weeks ago: Starting to flesh out the renderer 5 weeks ago: Added enums for slideshow styles, and a new attribute for the transition speed, so we can say how long images stay up, and how long they take to transition. 5 weeks ago: Make UniqueID optional, but keep uniqueIDString mandatory, the uniqueID is set on insert, and reset on fetch, so it's always gonna be there 5 weeks ago: Move code to render images to a texture to the top level texture unit class 5 weeks ago: Added all relevant shared source files for the slide show. 5 weeks ago: Keep screensaver on secondary display disconnect too. 5 weeks ago: Moved the screensaver to the app delegate, so we can screensave and connect to an external display automatically.