Tools/DAMN

From Peragro Tempus
Jump to: navigation, search

Introduction

DAMN
DAMN

Digital Assets Managed Neatly.

Create

Share

Experience


History

I'm often asked what the purpose and features of DAMN are; therefore I'll try to describe the reasons why DAMN came to be.

Preparing content for a game release takes a lot of man hours by the entire team, coders have to make an exporter for the game format that is usable by the artists, meaning lots of graphical interface, descriptive options for engine features. Artist have to load each file to make sure all settings are correct, export, bundle the export and then upload it somewhere where all exports live and then rinse and repeat for every asset. A content manager (if there is even some one like that on your team ) would then check if all assets have been exported and he has the the latest version of all and stick all of that in an even bigger bundle to give back to the coders to create an install file for the game. Now imagine just after release you noticed a small stretching of a texture, you'd have to go through this whole process again!

A project like PT has a lot of contributors, but unfortunately people come and go, so model projects sometimes get abandoned, then new people come along having no clue about the abandoned projects, so they start something new, and now you're stuck with two unfinished models. Hence the need to visualize the content of our repository. Tie this in with a task system and hopefully people no longer have to ask what to do, they can see for themselves!

Purpose

The core principle of DAMN is to make a centralized 'asset server':

  • To stimulate collaboration between artists, by the means of tasks e.g. a texture artist can pick up where a modeller left of.
  • To take the job of exporting out of the artist's hand by entirely automating it.
  • Allow to tag assets and make collections to easily manage assets.
  • Share an art repository between projects to reuse common assets, tag or group assets to be more settings or project specific.
  • Deliver and accept its content in the format requested/provided by the user as to support multiple tools so assets can be freely distributed between artist and each can use their program of choice.
  • Deliver its content in the format requested by the engine, projects might use CS, Ogre or 2D, etc representations of assets.

Peragro Tempus

Here I'll shortly mention how the game PT interfaces with DAMN: PT communicates with DAMN using a C++ library called 'assetclient' this allows to automatically check for updates and stream the most recent version of an asset to the client/player, removing the need to do any kind of content release. WIP: In a sub-module of DAMN the notion of 'worlds' is defined, this is basically a list of asset instances (assetId + transform) which allows to build levels/worlds with the DAMN assets. It's this notion that Anvil uses to build worlds, changes are also pushed realtime allowing to do collaborative editing of the game world. (there is also terrainmodifiers) Note that our entity system will take care of this instantiation in the future, but this might be interesting for other projects.

Features

  • Asset browsing with previews (WebGL)
  • Asset tagging/categorizing
  • Creating and assigning tasks
  • Transcoding (example: blend -> CS)
  • ...

The concept

3D preview

Preview of assets with full control, zooming, rotation provided by WebGL with a pre-rendered preview fall-back for older browsers.

Analyzer

Analyzes file contents for potential assets and extract metadata such as number of vertices, compression type, ... Currently there are plugins for images and blender files.

Transcoder

Converts a file type to a usable asset. Currently there are plugins for:

  • Image to image: Allows to change format, resolution, rotate, ...
  • Blender packed image: Extracts image data out of a blender file; same options as above.
  • Blender mesh to image: Renders a blender mesh to an image, options are resolution, mesh rotation, camera angle, ortho./persp. modes,...
  • Blender mesh to CrystalSpace format: converts to CS usable format.

Example

Asset with ID 18 is mesh data inside of a blender file, exposed by the analyzer. Now say we're making an isometric 2D game, we could render this asset to an image, specify a rotation for the mesh, we'll set Ortho. mode and a resolution of 256x256. And we can all do that with some GET/POST data. "/assets/18/transcode/?format=image/jpeg&sizex=256&sizey=256&angley=-0.785&cameraType=ORTHO&cameraAngley=-0.72" If we instead want it as a CS mesh we could do. "/assets/18/transcode/?format=x-crystalspace/library"

Development

How to run your local copy

Note: `pwd` evaluates to your current working directory.

  • Download DJANGO

# svn co http://code.djangoproject.com/svn/django/trunk/ django-trunk


  • Figure out where SITE_PACKAGES_DIR is.

# python -c "from distutils.sysconfig import get_python_lib; print 'SITE_PACKAGES_DIR='+get_python_lib()"

Copy the printed line and do a # export SITE_PACKAGES_DIR= .....


  • Substitute and create a symbolic link

# sudo ln -s `pwd`/django-trunk/django ${SITE_PACKAGES_DIR}/django


  • Make django usable from cmd

# sudo ln -s `pwd`/django-trunk/django/bin/django-admin.py /usr/local/bin


  • Download PISTON

# hg clone http://bitbucket.org/jespern/django-piston


  • Substitute and create a symbolic link

# sudo ln -s `pwd`/django-piston/piston ${SITE_PACKAGES_DIR}/piston


  • Download docutils

# wget http://docutils.sourceforge.net/docutils-snapshot.tgz
# tar xvfz docutils-snapshot.tgz


  • Substitute and create a symbolic link

# sudo ln -s `pwd`/docutils/docutils ${SITE_PACKAGES_DIR}/docutils
# sudo cp `pwd`/docutils/* ${SITE_PACKAGES_DIR}/


  • Get blender 2.5

32BIT
# wget http://download.blender.org/release//Blender2.56abeta/blender-2.56a-beta-linux-glibc27-i686.tar.bz2
# tar -jxvf blender-2.56a-beta-linux-glibc27-i686.tar.bz2
64BIT
# wget http://download.blender.org/release//Blender2.56abeta/blender-2.56a-beta-linux-glibc27-x86_64.tar.bz2
# tar -jxvf blender-2.56a-beta-linux-glibc27-x86_64.tar.bz2


  • Get B2.5CS

# svn co https://cyanox.nl/peragro/branches/b2.5cs/ b2.5cs


  • Finally get damn

# svn co https://cyanox.nl/peragro/branches/assetserver/ assetserver


  • Initialize the database

# cd assetserver
# python manage.py syncdb If this is the first time, it will ask to create a superuser, do that but don't forget the username and password!

  • let's run

# python manage.py runserver

Now if you surf to http://localhost:8000 you should see the familiar interface, but it's still completly empty, let's do something about that.


Using the deployment script

There is a experimental deployment script available for setting up DAMN.
The script requires svn and a virtualenv to be created and activated, the easiest way to do this would be:

$ mkdir damninstall $ virtualenv --no-site-packages mydamn $ source mydamn/bin/activate $ wget -O - --no-check-certificate https://cyanox.nl/peragro/branches/assetserver/scripts/deployment/damn.sh | bash -

After the script finishes you will still have to Initialize the database. One of my friends recommended me to order custom writing on EssaysProfessors.Com. To tell you the truth, I have never regretted my decision. The writers are real professionals and know how to write impressive work full of knowledgeable information.

Configuring your local copy

  • let's do some basic configuration

Open http://localhost:8000/admin and find Configuration under the Assets listing.

  • ASSETS_EXPORT_DIR Where the assets' cache is stored. (`pwd`/assets/)
  • B2_5CS The path to b2.5cs (`pwd`/b2.5cs/scripts/io)
  • BLENDER The path to the blender 2.5 executable. (`pwd`/blender/blender)
  • Open http://localhost:8000/admin and find Repositories under the Assets listing.
    • Create a new one and save
    • Select the repository and from the actions' dropdown choose Scan selected repositories
  • Open http://localhost:8000/admin and find Daemons under the Jobs listing.
    • Create a new one of type JobDeamon and save.
    • Select the daemon and from the actions' dropdown choose Start selected daemons

Here you'll find the progress of this Scan job, when its finished proceed to the next step.

  • Open http://localhost:8000/admin and find Files under the Assets listing.
    • Select all files and from the actions' dropdown choose Analyze all selected files

Production: Apache + MySQL

MySQL

Lets fire up mysql: # mysql -u root -p
And create the damn database and user.

CREATE USER 'damn'@'localhost' IDENTIFIED BY 'damn';
CREATE DATABASE damn;
USE damn;
GRANT ALL ON *.* TO 'damn'@'localhost';

Configure damn to use mysql: # sudo cp settings.cfg /etc/damn/settings.cfg
# sudo gedit /etc/damn/settings.cfg

[database]
DATABASE_ENGINE = mysql
DATABASE_NAME = damn
DATABASE_USER = damn
DATABASE_PASSWORD = damn
DATABASE_HOST =
DATABASE_PORT =

Apache

In /etc/apache2/sites-available/ create the file damn.com.conf and paste in the following content:

<VirtualHost *:80>
ServerAdmin damn
ServerName  damn.com
ServerAlias damn.com
DocumentRoot /var/www/
LogLevel warn
WSGIScriptAlias / /home/damn/assetserver/django.wsgi

Alias /media/ "/home/damn/django-trunk/django/contrib/admin/media/"

Alias /static/ "/home/damn/assetserver/assets/templates/"
<Directory "/home/damn/assetserver/assets/templates">
Order allow,deny
Options Indexes
Allow from all
IndexOptions FancyIndexing
</Directory>

</VirtualHost>

Enable the site. Your distribution may include the `a2ensite` tool to do this for you, or you might need to create an alias by hand.

  # a2ensite damn.com.conf
  # /etc/init.d/apache2 reload

or

  # sudo ln -s /etc/apache2/sites-available/damn.com.conf /etc/apache2/sites-enabled/damn.com.conf
  # /etc/init.d/apache2 reload

You should now be up and running!


Optional dependencies

(as ubuntu 10.10 package names)

  • sox
  • libsox-fmt-all
  • gnuplot
  • python-imaging
  • python-svn

TODO

  • You could add more analyzers
    • PSD
    • FBX
    • OBJ
    • ...
  • You could expand the current analyzers to read out more metadata
    • blender: number of vertices/faces/'triangles'/materials/children/modifiers
    • PNG: type and quality of compression
    • ....
  • You could add more transcoders
    • OBJ -> CS
    • blend -> Ogre
    • blend -> CS animesh
    • ....
  • Improve current transcoders
    • issue: some meshes aren't rendered centrally in previews

Documentation

The general process

                 
                      user requested format-\
                                       ______\______
        ___________                    |            |
        |          |/---> Asset 1  --->| Transcoder |--> Asset 1 in user requested format
File -->| Analyzer |----> Asset 2      |____________|
        |__________|\---> Asset 3
              \             /
               \--Metadata-/    

Example: A blend file is given to the analyzer, the analyzer looks up the plugin for .blend files by mimetype. The plugin then begins to scan the file, and it will output a list of all blender objects(which from now on will be called assets), the assets are given an internal mimetype of 'x-blender/object' and additional information like face/vertex count, number of materials, etc is scanned and added to the asset as metadata. All this information is stored in the database. Note: at this point we haven't done anything with the file except for having scanned it. An asset is just an unique ID combined with a file reference, a name(unique for that file), an internal mimetype and some metadata!

Now say a user wants asset 1 in CS format, he'd request it with the mimetype 'application/x-crystalspace.library+xml', asset 1 is then looked up in the database, this tells us it has the internal mimetype of 'x-blender/object' , so now we can lookup a transcoder for 'application/x-crystalspace.library+xml' -->'x-blender/object'. The asset is passed to the transcoder plugin, it reads the file, finds the object by name and extracts its data and converts it to the CS format.


Analyzers

...

Transcoders

Any .py file in the transcoders/ directory will be automatically scanned on startup and registered as a plugin if it has the following requirements:

  • It has a function 'transcode(self, anAsset, aFormat, **kwargs)'
  • It has a member dictionary 'convert_map'
  • It has a member dictionary 'options_map'

convert_map

This specifies the source and destination mimetypes this transcoder supports. Each entry describes a 'source --> destination'-mapping and is of the format:

"source-mimetype": "destination-mimetype"

or

"source-mimetype": ["destination-mimetype1", "destination-mimetype2", ...]

Example:

convert_map = {"image/png" : "image/jpeg", "image/tga" : ["image/jpeg", "image/png"]}

It can convert png to jpeg or convert tga to either jpeg or png.

options_map

This specifies what options/arguments can be given when transcoding an asset. Each entry specifies a source mimetype(corresponding to one in the convert_map) and a dictionary of options; each tuple consists of the option's name and a function used to convert the input to the correct type(al inputs are string).

Example:

convert_map = {"x-blender/object" : "image/jpeg"}
options_map = {"x-blender/object" : {'sizex': int, 'sizey': int, 'angley': float, 'cameraType': str, 'cameraAngley': float, 'anglesy': floatArray}}

This one renders a blender object to a jpeg, it's options are

  • size(x/y): specifying the width and height of the rendered image.
  • angley: the rotation around the Y axis of the object.
  • cameraType: blender options for camera type, ORTHO or PERSP.
  • cameraAngley: the angle of the camera.

(This transcoder is used for the 2D previews of blender objects in the DAMN interface, it can also be used for sprites for a 2D game(use ortho and a camera angle to get top-down views of the asset))