Cinelerra Server

From Deptford.TV

Jump to: navigation, search

Contents

[edit] System Overview

This page describes the online parts of the system for collaborative editing. ie. gives further detail to the blue parts of the Workflow overview diagram.


The online platform does 3 things:

  1. Provides a repository and annotated archive of raw materials gathered by project participants.
  2. Allows project participants to share and annotate edited versions that use that material.
  3. Enables collaborators to download packages of files that will allow them to immediately edit other peoples edited "versions" on their own local machine using cinelerra, and subsequently upload their own revision with a log message.

[edit] Cinelerra Server

This page describes how to set up "Cinelerra Server" in order to be able to collaboratively edit with [cinelerraCV]. The raw unedited materials and [xml] project files produced by cinelerra are held in a customised installation of the CMS/application framework [drupal].

The basic setup consists of:

  1. A standard [LAMP] web and mail server (outgoing mail only required).
  2. An installation of drupal6 as described below, plus two custom modules, several custom drupal content types and several custom views. Currently available from our own svn repostitory.
  3. A chrooted "dropbox" area for participants to upload files via sftp. This allows robust and efficient transfer of files to the server. Once uploaded the files are transformed into attachments to unique drupal nodes via custom extensions to the drupal "Media Mover" module.

The documentation below describes the setup of the case study http://edit.deptford.tv and the code needed.

[edit] server setup

  1. This installation follows the [debian lenny server] install described on how to forge. But with a [LVM] [Raid-1] system. You follow the instructions until step 3 "Base system". Then when coming to partition disk you don't choose the option "guided" but "manually" as described bellow (of course only if you wish to have a LVM Raid-1 system as well, else simply continue with the "guided" option).
  2. how to set up LVM Raid1
  3. then continue with step 3 where it says "Now the base system is installed:" until step 6 "configure the network"
  4. how to configure the network
  5. then continue as described with [with /etc/init.d/networking restart]
  6. stop with step number 9 & follow the instructions below
  7. add users and disable root logins
  8. how to set up AMP Apache MySQL PhP
  9. how to test AMP
  10. how to set up mail server

[edit] drupal setup

We will be working on a installation profile for cinelerra server that we can package and publish. However, for now, you wish to set up your own instance of cinelerra server please contact us and we will provide you with access files that you need including a database dump with all our configurations but empty of content.


1. Drupal Core and contributed modules

Follow instructions for a standard drupal 6 installation. Eg. [[1]] and [[2]]. Ensure that the "files" directory is correctly set up as this is particularly important in our setup.

You will need the following contributed modules which you are recommeded to install in your sites/all/modules directory:

essential

  • cck
  • content_profile
  • date
  • filefield
  • getid3
  • imageapi
  • imagecache
  • imagecache_profiles
  • imagefield
  • link
  • location
  • views
  • checkbox_validate
  • diff
  • legal
  • media_mover


recommended

  • admin_menu
  • jq
  • jquery_media
  • tagadelic


Notes on the specific configuration of these tbc, except for media mover in which case see dropbox configuration section below.


2. DTV Custom modules and set up

We have two custom modules which are available from our svn repository (ask us for access) and/or can be viewed at [[3]]. mm_dtv is an extension of the media mover contributed module which builds drupal nodes from video files; dtv_utils mainly deals with extracting and keeping track of asset files contained in the xml files of our different versions.

There is also a database dump of an empty of content and users database which you can import instead of the usual drupal empty database. This file is in trunk and is called dtv_empty.sql.gz. The admin user is already set with password temp. Please change the admin password immediately on installation.

You are recommended to install them in either sites/<path to your drupal setup>/modules (see drupal multi-site setup), or sites/default/modules. You can create these directories and then do an svn checkout of trunk directly into them.

You should create your files directory (ensure it is writeable by the webserver) in either sites/default or sites/<path to your drupal files>. In the "blank" database the files directory is set to sites/default/files but you can change this.


Project content types are set to "create new revision" if edited, thus creating a wiki-like trace of all changes made to a particular version. The drupal contributed diff module is currently used to show the differences between versions, but we hope to develop a more accessible storyboard style output soon with the storyboader already developed in Scripts_and_Prototypes as a base.

At the current time no customisations have been made at the theme level so you are free to try out any drupal theme you wish, custom or otherwise, with our system.

TODO: a fully-fledged installation profile for the system.

[edit] dropbox and media mover setup

Audio and image files are uploaded into drupal via the web interface by simply creating a new audio or image node.

However, as we are usually dealing with very large files for video, we have created a custom setup for uploading video files and importing them into drupal nodes.

In this setup, each user has a "dropbox" accessible to them via sftp where they may upload video files. This makes a secure and robust way for participants to transfer a number large files using an sftp client that can be left to run, for example, overnight.

Each individiual user dropbox is identified by the users drupal uid. We have created some custom extensions for the drupal contributed media mover module to move files from the dropbox into the drupal platform. When our media mover configuration is run, the following actions take place (grouped according to the media mover api):

  • harvest (ie. identify files to be processed)
    • list the files in all the users dropboxes. Build a list of these, excluding files that have been processed before (by comparison with the mm_files table).

This means that files do not have to be deleted by individual users and no duplication should take place.

  • storage
    • check each filename is unique and generate an informative error if not
    • create a new node for each valid file found
    • attach the video file to that node using a filefield cck field, save it to a correct location within your drupal installations designated files/ directory, according to the setup n the cck field. Additionally, our module places the video file a subdirectory based on the year and month that the file was created (uploaded to server), to create manageable sized directories. (the subdirectory is created if it does not already exist).
    • set the owner of the node to the uid of the dropbox in which it was found
    • set the status of the node to unpublished (it will be set to published when the node has sufficient extra information attached to it)
    • in the case of mp4 files (which we are not using as of writing), metadata added via id3 tags are written into the relevant fields in the node.
    • generate an informative log message
  • complete
    • ensure that the node creation and file move has been recorded correctly in the media mover table


chroot setup

The dropbox is created and secured using the package scponly in a chroot. This means that user accounts for the dropbox do not have any access to the main server system. They cannot access the main server filesystem, log in with a shell or run any commands. A seperate set of libraries run the services in the chroot jail so even if the chroot is compromised in some way, the main server system should keep intact.

Scponly is available as a debian package. see: [[4]]. and [[5]] for help.

Within our single chroot we have set up a "home" directory for each user, which is named according to their drupal userid (number) rather then their login name. This means we can identify the user when we build nodes from the files in the dropbox.

To create dropbox users, a shell script has been created that must be run as root directly on the server. Note that the username and password of dropbox users is currently COMPLETELY UNRELATED to the drupal users although you need to know the drupal numeric uid of the user. (A job for phase II will be to integrate drupal and dropbox users).

Instructions for creating a dropbox user:
  1. Create the user in drupal, if necessary
  2. Get the drupal user id number, for example by navigating to the users personal page, and get the number at the end of the url, eg: http://edit.deptford.tv/user/6, or use users list page from the admin menu, and get the uid from the links to each users page.
  3. Log in to edit.deptford.tv
  4. cd to /root
  5. run the script called add_drupal_user.sh with 2 arguments. The first should be the username (not necessarily the drupal user name eg. you may need to lowercase everything to make it a valid name). The second argument should be the drupal uid.
  6. You will be asked for a password. Make sure you note it to send to the user as it will NOT be the drupal password. Make sure it is a secure password. Although users are chrooted, we do need to make sure we're not too vulnerable.
Example usage:
sudo ./add_drupal_user.sh lisa 6

The script ensures that the users shell is scponly, that the home directory is created inside the chroot and is named according to the users' drupal numeric id so that files uploaded there have the correct ownership.

A quick way to see which users are already dropbox users is to have a look at the passwd file in the chroot, they'll be listed there: To change users passwords use the passwd command as with any normal user.

A cleanup script for the dropbox, that deletes all .ogv or .mp4 files in users dropboxes that are more than 24 hours old, is in /root/scripts. its called dropbox_cleanup.sh and it keeps a record of all the deletions file paths in a text file: /home/drop1/FILE_DELETIONS_RECORD.txt. The media mover list of all files transferred is untouched by this but it shouldn't matter.

If a user needs to run the same file through media mover again because of some kind error, they must 1. delete the node created eg. by finding it at content->list, AND 2. must delete the entry in the media mover list of files at media_mover->dtv_uploader->List files.



NOTE that for the media mover configurations to work it is necessary for php the php open_basedir directive to be set to access the dropbox (aswell as tmp/ and the document root of your drupal site). It is strongly suggested you set this in the virtual host configuration of the domain you run your site from, and not for your entire webserver configuration.

Contact us if you require more information on this setup. Its a bit non-standard. Also contact us if you can spot any gaping security holes in what we've done ;)

[edit] code

This section is not essential reading for setting up the system, but describes the customisations made in code, in addition to the media mover custom functions described above.

Our code is available via our svn repository but because it is in pre-alpha stage and needs a good tidy up, including conformance to drupal coding standards, its not published here yet. Please email if you would like access to the code.

implementation of hook nodeapi:

  • show the download archive button on view of project nodes, if perms allow. This button will create for user download an archive of the currently viewed revision of a project node which will contain:
    • all the required assets (unless they have been downloaded by this user for this project previously).
    • the xml project file you will use on your local machine to launch the edit.
    • a text file called assets.txt that provides you with a nice offline listing of all the data related to the assets you have downloaded. This links to code largely already created for an earlier prototype of this project see Scripts_and_Prototypes.
    • We also track a particular users downloads for each project via a custom table that relates uid, project node id and asset file node id. This is to avoid repeated archiving and downloading of large files.
  • validate for unique filenames on all node types
  • add links to assets using <video> or <audio> tags so that an embedded ogg player shows, if the browser allows. (This should be at theme level but its here for now to be theme-agnostic).
  • add nice links for downloading asset files, if permissions allow.
  • when video, audio and image nodes are saved, check for important fields and taxonomy terms being filled in. If they are, the node becomes published.

implementation of hook perm

  • defines permissions for downloading project archives
  • defines permissions for downloading clips


implementation of hook menu

  • create a tab on each users page so they can clear their download history

Implementation of Hook Cron

  • cleans up the temp location of archives so we are not cluttered by these large files

Implementation of Hook Form Alter

  • validates the xml in the uploaded project file, by checking for incorrect (non-relative) file paths and assets that are not uploaded to the platform yet.
  • on submit of a project file, reads the contents of the uploaded file xml file into the xml_content field of the node.
  • on submit of a project file, creates nodereference fields for assets that appear in the project to keep a running list of raw materials in each edit.
  • unsets file upload form elements for assets, so no new assets can be uploaded for a particular node.
Personal tools