c - Thread pool where the tasks are relatively large -

i have created thread pool in c. each thread in pool performs same function. several producer threads put fresh data pool's queue using standard mutex/cond method.

the fresh data relatively large, , amount of processing has performed can take quite awhile. whenever have seen implemented workers lock queue while either copy required data or perform requested task. in case either of these operations may take awhile, , during time other threads blocked accessing queue.

how should go announcing particular thread has taken task data associated task still being used? practical add sort of 'in process' flag queue , let worker thread unlock queue while works?

revise queuing structure what's queued pointer data processed.

when thread needs grab job, grabs mutex, gets next pointer task performed, possibly zaps in-queue copy of pointer or otherwise makes sure other threads won't process processing, , releases mutex (or signals condition or whatever). access big chunk of data via pointer; 1 thread has pointer; cleans when done, operates secure in knowledge no other thread working on same data.

so, finesse problem of big chunks of data not using them in queue — use little chunks of data, aka pointer big chunk.

currently queue queue of pointers. however, malloc finite number of buffers @ beginning of program. correct me if wrong, suggesting instead have producer threads malloc memory required, , have workers free when finish?

i'm not suggesting have producers use malloc() , consumers use free().

what i'm suggesting organize queue there no question of either producer or consumer needing lock extended period of time. however, if queue queue of pointers, don't understand how you've got problems in first place.

it comes down terminology — great causer of confusion.

what i'm trying suggest there queue of 'tasks waiting consumed'. sometimes, queue empty; consumer threads wait on condition 'queue not empty', , when producer adds task queue, 1 of waiting consumers woken , take on running new task. consumer's initial step remove task 'waiting consumed' queue.

the information on queue must sufficient identify needs doing — may pointer 'task description', contains further pointers data operated on stored. removing 'task description' queue should (must be) quick, simple operation (protected mutexes , conditions). given task description not accessed multiple threads concurrently data points @ not accessed other threads (in general). if there data shared between threads, concurrent access shared data must coordinated usual.

but key design point consumer thread spends minimal time blocking either other consumer threads or producer threads while manipulating queue. gains access queue, removes first item on queue, , releases access queue. gets on processing task needs done — no interference producers or other consumers.

likewise, producer thread prepares task description , ensures relevant (pre-allocated) buffer used, etc — handling acquisition of buffer carefully, etc. when task description ready, producer spends short time acquiring access queue, adding task description queue, releasing access queue, signalling 'queue not empty' condition.