Integrating daemons, PHP with multi-threading

Portal Home > Knowledgebase > Articles Database > Integrating daemons, PHP with multi-threading

Posted by Niklas, 05-18-2010, 05:13 PM
Before I go into detail of what I was planning on doing, probably better to explain what I want it to do, in case someone has better ideas how to do it. I am developing an application in PHP/MySQL which allows videos to be uploaded, after which it adds them into a queue for encoding. The encoding itself is performed by mencoder, and there isn't any problems regarding that matter. The problem I am facing is that these videos need to be queued (can't have more than one encoding process running at a time), and they need to have support for multi-threading. Additionally, using unix-only supported functionality isn't an option which rules out php pcntl (which otherwise would been quite ideal, except it doesn't support multi-threading either). The solution I had planned was to have the php application write information files with details such as the name for the input file, output file, the bitrate, fps and related information. I would then have a daemon fetching the oldest info entry and get the information from there which it would then use for the encoding process using mencoder, after which it would delete the info file (so it wouldn't re-encode the file right on the next loop). Finally, the php application would using crons check a folder regularly if there are new jobs complete there after which it would apply them to the database. Now, everything related to the php application is clear and there aren't any problems there, but I've never written a daemon before, so I am not sure what is the best approach here. Additionally, is it wise to have the daemon read the information from files instead of directly from the database? If not, what format would be the easiest for the daemon the interpret? XML? What language would be most ideal for the daemon? Are there any good skeleton daemons anyone could recommend, that could be used for building my functionality on top of it?
Posted by sasha, 05-18-2010, 10:09 PM
I am PHP fan, but I would not chose PHP for this job. If you do not want to go too low level, I would recommend looking in python for this job.
Posted by The Universes, 05-18-2010, 11:57 PM
You can definitely write this backend daemon in PHP if you are the most comfortable with it. I would just store the videos that need to be encoded into a table (database) and have a cron run a PHP script every 1 min. That PHP script would fetch the (single) oldest unconverted video and mark in the database its processing it (some sort of a status column) and call mencoder. Once the conversion is over, the script would update the table and mark that file as finished and do whatever else is necessary. Anyways, this is just a suggestion if you plan to use PHP. This obviously doesn't have perfect multi-threading, but the delay is minimal since it runs every minute and each script processes 1 video. Last edited by The Universes; 05-19-2010 at 12:05 AM. Reason: typos
Posted by mattle, 05-19-2010, 08:31 AM
There's no need to use cron for this. Most daemons simply revolve around a loop like: You simply start the process in the background: Or, if you want it to survive a reboot, you can launch it from /etc/rc.local. If you want to get really fancy (and you're using a RH variant), you can write a script in /etc/init.d and add it as a service using chkconfig. I do agree with TheUniverses that you want to use a database to store the video's metadata. I don't know that you'll really want to store the videos themselves in a database. I generally don't store binaries in a db unless I've got a really good reason. In this scenario, I'd opt for storing them in the filesystem and having a field in the database for the file path.
Posted by tim2718281, 05-19-2010, 01:42 PM
Language doesn't matter; use bash, cshell, php ... whatever scripting language you are comfortable with. It's really a matter of organising your thoughts on exactly what you want to happen to the data, then writing the glue to make it so. Personally, I always avoid using a database for this kind of task; instead I use file system files. The reason is, if something goes wrong in the mniddle of the night, it seems easier for whoever is on call to determine the state from looking at file lists than by accessing a database. One tip; move things from one directory to another to control state changes. For example, files should be uploaded to a directory whose name means "upload_in_progress". When the upload completes, the upload process should move the file to a directory whose name means "upload_complete". If you use cron jobs, the job should look in the "upload_complete"; this way, the cron job will never see an upload that is still in progress. If it finds anything, the cron job should create a temporary directory whose name means "cron_job_working_on_it" and move the file there. You could choose to have more than one cron job running at a time; each will create its own temporary directory and attempt to move the file there. If two cron jobs attempt to move the same file, only one will succeed. So the cron job needs to check the results of its move command, to see if it was the winner. OK, so much for synchronisation. When the cron job completes processing a file, it will move the output from the temporary directory to the desired location, and also move the input to a directory whose name means "conversion_completed". The cron job can then optionally look for more work - it depends on whether you want to control the maximum conversion rate using cron - and if none, clean up the temporary directory and exit. If things go wrong you can see the current state by browsing the file system; you will be able to see what conversions completed, what conversions are in the input queue, and which conversions are currently in progress. if you make the temporary directory name include the cron job process id, you'll be able to see what processes are running for which conversions. You may detect my preference for control scheduling by cron; the reason is simple - that's what cron was designed to do, and it does it reasonably well. And if a conversion hangs, it doesn't affect other conversions - a new cron job will fire in a while. One thing I have found helpful in the past is to have these automated file-creation programs check available disk space and simply exit with a message if there is less than some specified value.
Posted by HostTantra, 05-20-2010, 01:00 AM
Check out phpmultithreaddaemon.blogspot.com. But its better to use python for the job. Check out SuperVisorD which is a python daemons supervisor, it brings up the daemon processes under it whenever the custom daemon goes down.
Posted by 4umfreak, 05-20-2010, 10:01 AM
It's interesting to get everyones different perspectives! I would have suggested a cron triggered shell script that converts the files and moves them through the same directory structure mentioned earlier. If that's working but you need more files processed, then modify it to run like a daemon.
Posted by HostTantra, 05-20-2010, 07:15 PM
Crons have their set of problems. Check out http://kevin.vanzonneveld.net/techbl...aemons_in_php/, the segment Deamons vs Cronjobs explains some of the advantages of Daemons over Cronjobs.

Add to Favourites Print this Article