Fixing Broken Functionality

From approximately 5-October until 23-November, in-browser video playback in OMwiki was completely broken.  Visitors were greeted with a “Video playback is broken at the moment. :-/” top-posted message, which leads to poor first impressions and undermines our overall credibility.  So, what happened and how can it be avoided in the future?

Click here to skip the tech breakdown.

What happened?
All video currently hosted on OpenMeetings.org is encoded in the Ogg Theora format and is stored in the /archives subdirectory.  Currently, there is no content delivery network (CDN) in place, so all video originates from a single server.  This server utilizes oggz-chop, which is part of the oggz toolset — it’s one of several programs that manipulate Ogg files in helpful ways.  oggz-chop is a binary program that returns the very beginning of an Ogg Theora file proceeded immediately by the section of interest indicated by a query string appended to the URL request (/archives/video.ogv?query_string).  The query string contains the requested start/end time, and makes possible user-friendly jump-to-this-time URLs.  Thus, we achieve intelligent seeking.

Because oggz-chop is a binary program, it is sensitive to changes in the host operating system.  Sometime early October, our hosting provider migrated all shared hosting plans to a new compute cluster and operating system.  The migration was unannounced as there is a general expectation that shared hosting plans will not run executable binaries; most customers would not have noticed a difference.  Compiling natively on both the old and new clusters is disabled for security.

How was it fixed?
The new operating system lacked the libogg.so shared library, a mandatory dependency.  Both libogg and oggz-chop where compiled on a binary-compatible machine and the resulting binaries were copied over.  This is a mostly-blind trail-and-error process.

I unsuccessfully attempted to create a single binary that contained both libogg and oggz-chop via static linking.  I found alternative route by including a runtime search path (export CFLAGS='-Wl,-rpath=/home/openmeet') such that oggz-chop could locate libogg.so in a non-system directory.  Tweak some variable names in cgi.c, and voilĂ !

#define DOCUMENT_ROOT “/home/openmeet/public_html”
. . .
getenv (“PATH_INFO”); –change-to–> getenv (“REDIRECT_URL”);

[Post truncated for brevity 11-June-2012]

5 thoughts on “Fixing Broken Functionality

  1. GChriss Post author

    I’ll also note that there are a large number of videos that don’t play as the local copies in the /archives directory were inadvertently removed. It makes better sense to fix this issue as soon as hosting upgrades and a content delivery network are in place.

  2. Luther Knight

    Another term for shared hosting is virtual hosting. It is important to differentiate between “shared/virtual hosting” and ” VPS hosting .” Shared hosting usually entails many dozens, hundreds, or even thousands of websites intermingled on one server device. VPS hosting usually consists of less than one dozen websites (and their applications) residing on an individual server device with certain electronic partitions demarking each site into reserved zones on that server device.

    1. GChriss Post author

      At the moment OpenMeetings.org indeed runs on shared (not VPS) hosting, although resource allocation has been reasonably OK. -G

      1. Elton

        I’ll also note that there are a large number of veiods that don’t play as the local copies in the /archives directory were inadvertently removed. It makes better sense to fix this issue as soon as hosting upgrades and a content delivery network are in place.

        1. GChriss Post author

          I’ve restored the missing recordings in the ‘/archives‘ directory and included a Bash script used to download each from the Internet Archive. This process will be automated into a Cron job but that’s unimplemented at the moment.

Comments are closed.