Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

After setting up queue listener to process uploaded videos with FFMPEG, I've come back to the server several times to find that MySql has stopped running. I checked drive space and it's about 77% used (43G out of 60G).

Here's my code in case it's useful:

public function fire($job, $data)
{
$data = json_decode($data['transcoding_message'], true);
$output_directory = '/home/ubuntu/transcodes/';
$amazon_array = array();

$s3 = AWS::get('s3');


//execute main transcoding thread

$cmd = 'sudo ffmpeg -i ' . $data['temp_file_url'] . ' -y -vcodec libx264 -tune zerolatency -movflags faststart -crf 20 -profile:v main -level:v 3.1 -acodec libfdk_aac -b:a 256k ' . $output_directory. $data['temp_file_key'] . '_HQ.mp4 -vcodec libx264 -s ' . $sq_width . 'x' . $sq_height . ' -tune zerolatency -movflags faststart -crf 25 -profile:v main -level:v 3.1 -acodec libfdk_aac -b:a 256k ' . $output_directory. $data['temp_file_key'] . '_SQ.mp4 -ss ' . $seek_half . ' -f image2 -vf scale=iw/2:-1 -vframes 1 ' . $output_directory. $data['temp_file_key'] . '_thumb.jpg';

exec($cmd." 2>&1", $out, $ret);


if ($ret)
{

    Log::error($cmd);
    echo 'Processing error' . PHP_EOL;
    //there was a problem
    return;
}

else
{
    //setup file urls
echo 'about to move files';

$hq_url = $this->bucket_root . $data['user_id'] . '/' . $data['temp_file_key'] . '_HQ.mp4';
$sq_url = $this->bucket_root . $data['user_id'] . '/' . $data['temp_file_key'] . '_SQ.mp4';;
$thumb_url = $this->bucket_root . $data['user_id'] . '/' . $data['temp_file_key'] . '_thumb.jpg';

$amazon_array['video_hq_url'] = $data['temp_file_key'] . '_HQ.mp4';
$amazon_array['video_sq_url'] = $data['temp_file_key'] . '_SQ.mp4';
$amazon_array['video_thumb_url'] = $data['temp_file_key'] . '_thumb.jpg';

        //copy from temp to permanent

foreach ($amazon_array as $k => $f)
{
    $uploader = UploadBuilder::newInstance()
        ->setClient($s3)
        ->setSource($output_directory.$f)
        ->setBucket($this->bucket)
        ->setKey('users/' . $data['user_id'] . '/' . $f)
        ->setConcurrency(10)
        ->setOption('ACL', 'public-read')
        ->build();

    $uploader->getEventDispatcher()->addListener(
        'multipart_upload.after_part_upload',
        function($event) use ($f) {
            // Do whatever you want
        }
    );

    try {
        $uploader->upload();
        echo "{$k} => Upload complete.\n" . PHP_EOL;

        DB::table('items')->where('id', $data['item_id'])->update(array($k => $this->bucket_root. $data['user_id'] . '/' .$f, 'deleted_at' => NULL));
        //delete local

        unlink($output_directory.$f);

        unset($uploader);
    } catch (MultipartUploadException $e) {
        $uploader->abort();
        echo "{$k} => Upload failed.\n" . PHP_EOL;
        continue;
    }
}

    //write to database

        DB::table('archives_items')->where('id', $data['archive_item_id'])->update(array('deleted_at' => NULL));
        DB::connection('mysql3')->table('video_processing')->where('id', $data['id'])->update(array('finished_processing' => 1));

    //delete files

    //delete s3
    $s3->deleteObject(
        array(
            'Bucket' => $this->temp_bucket,
            'Key' => $data['file_name']
        )
    );
    echo $data['temp_file_url'] . '=>' . " deleted from temp bucket.\n" . PHP_EOL;
    DB::connection('mysql3')->table('video_processing')->where('id', $data['id'])->update(array('deleted_at' => \Carbon\Carbon::now()));

}
    $job->delete();

    // end of processing uploaded video
    }


else
{
    return;
}

Any ideas as to why MySql would die like that?

ERROR 2002 (HY000): Can't connect to local MySQL server through socket     '/var/run/mysqld/mysqld.sock' (111)

I want to mention that the php artisan queue:listen command is being triggered via Supervisor and that I have 4 running concurrent processes.

Error log:

140613  5:37:52 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
140613  5:37:52 [Note] Plugin 'FEDERATED' is disabled.
140613  5:37:52 InnoDB: The InnoDB memory heap is disabled
140613  5:37:52 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140613  5:37:52 InnoDB: Compressed tables use zlib 1.2.3.4
140613  5:37:52 InnoDB: Initializing buffer pool, size = 128.0M
140613  5:37:52 InnoDB: Completed initialization of buffer pool
140613  5:37:52 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
140613  5:37:52  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
140613  5:37:53  InnoDB: Waiting for the background threads to start
140613  5:37:54 InnoDB: 5.5.37 started; log sequence number 2922980986
140613  5:37:54 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
140613  5:37:54 [Note]   - '127.0.0.1' resolves to '127.0.0.1';
140613  5:37:54 [Note] Server socket created on IP: '127.0.0.1'.
140613  5:37:55 [Note] Event Scheduler: Loaded 0 events
140613  5:37:55 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.37-0ubuntu0.12.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
140613  5:38:10 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
140613  5:38:10 [Note] Plugin 'FEDERATED' is disabled.
140613  5:38:10 InnoDB: The InnoDB memory heap is disabled
140613  5:38:10 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140613  5:38:10 InnoDB: Compressed tables use zlib 1.2.3.4
140613  5:38:10 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
140613  5:38:10 InnoDB: Completed initialization of buffer pool
140613  5:38:10 InnoDB: Fatal error: cannot allocate memory for the buffer pool
140613  5:38:10 [ERROR] Plugin 'InnoDB' init function returned error.
140613  5:38:10 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140613  5:38:10 [ERROR] Unknown/unsupported storage engine: InnoDB
140613  5:38:10 [ERROR] Aborting
140613  5:38:10 [Note] /usr/sbin/mysqld: Shutdown complete
share|improve this question
2  
Why did MySQL stop running? Check its logs. –  Michael Hampton Jun 13 at 16:40

1 Answer 1

up vote 0 down vote accepted

InnoDB: mmap(137363456 bytes) failed; errno 12

140613 5:38:10 InnoDB: Fatal error: cannot allocate memory for the buffer pool

Your system ran out of memory error 12 is the kernel signaling ENOMEM.

share|improve this answer
    
Since the command php artisan queue:work --queue=sqs.us-west-2.amazonaws.com/587107866605/… --delay=0 --memory=128 --sleep=3 --tries=0 --env=local was executed, would this queue process failing bring down MySql with it? –  egekhter Jun 13 at 18:12
    
It looks like your system ran out of memory is all I know. –  Iain Jun 13 at 18:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.