Add tests for all uncovered BatchEnqueueContext error paths: nil task,
empty/blank typename, invalid options, group rejection, unique rejection,
and broker-down (sleeping broker). Add pipeline error test for rdb.BatchEnqueue
via cancelled context, and TestTaskOptions/TestTaskOptionsNil for the
new Options() accessor on Task.
Add comprehensive doc comments to BatchEnqueueContext, the Broker
interface method, and the RDB implementation explaining that the batch
uses a Redis pipeline (not MULTI/EXEC), so partial success is possible
and individual Lua scripts are atomic but the batch is not.
Fix a bug where Script.Run inside a pipeline only sends EVALSHA without
the automatic EVAL fallback that non-pipeline calls get. On a fresh
Redis (or after SCRIPT FLUSH), this caused NOSCRIPT errors for every
pipeline-batched script invocation. The fix preloads the required Lua
scripts before building the pipeline.
Also roll back the in-memory queuesPublished cache when the pipeline
fails, preventing stale entries from suppressing future SADD calls.
Add TestCancelationPubSubReceiveError to verify that when
Receive() fails in CancelationPubSub(), an error is returned
and the pubsub connection is not leaked.
This provides test coverage for the pubsub.Close() fix that
was missing in the previous commit.
Bahtya
When redis.Subscribe succeeds but Receive() fails, the pubsub
connection was not being closed before returning the error. This caused
the subscriber goroutine to leak a Redis connection on each retry
iteration, eventually exhausting the connection pool.
Fixes#1095
BatchEnqueueContext had a time comparison bug where `now` was captured
before the loop but `processAt` was set to time.Now() inside
composeOptions during each iteration, causing all immediate tasks to be
incorrectly classified as scheduled and rejected.
Fix: move `now` capture inside the loop, after composeOptions.
Additionally, extend BatchEnqueueContext to support scheduled tasks in
the same pipeline. Tasks with a future ProcessAt are now routed to
scheduleCmd (ZADD to scheduled set) instead of being rejected. Only
unique and group tasks remain unsupported.
Changes:
- Add BatchEnqueueItem type pairing TaskMessage with optional ProcessAt
- Update Broker interface, RDB, and testbroker to use BatchEnqueueItem
- Route immediate tasks to enqueueCmd, scheduled tasks to scheduleCmd
- Return correct TaskState (Pending vs Scheduled) in results
- Add tests for immediate, scheduled, and mixed batch scenarios
Adds BatchEnqueue to the Broker interface and RDB implementation that
sends multiple enqueueCmd Lua script invocations in a single Redis
pipeline round-trip. Also adds BatchEnqueueContext to the Client as
the public API, returning per-task results for partial-success handling.
Ref: hibiken/asynq#1069
MEMORY USAGE returns nil for keys that no longer exist (e.g., expired
or deleted task keys). In Lua, nil is converted to false (a boolean).
The script then attempts arithmetic on this boolean value, causing:
ERR user_script:30: attempt to perform arithmetic on local 'bytes'
(a boolean value)
This breaks the /api/queues endpoint in asynqmon, showing "Could not
retrieve queues live data" in the UI.
The fix adds nil guards around all three MEMORY USAGE calls on task
keys, and a divide-by-zero guard on agg_task_sample_size.
Tested in production with Redis 7.2 and asynq v0.25.1 worker.
Fixes#728
Related to #901
* Improve performance of enqueueing tasks
Add an in-memory cache to keep track of all the queues. Use this cache
to avoid sending an SADD since after the first call, that extra network
call isn't necessary.
The cache will expire every 10 secs so for cases where the queue is
deleted from asynq:queues set, it can be added again next time a task is
enqueued to it.
* Use sync.Map to simplify the conditional SADD
* Cleanup queuePublished in RemoveQueue
---------
Co-authored-by: Yousif <753751+yousifh@users.noreply.github.com>
* feature: configurable janitor interval and deletion batch size
* warn user when they set a big number of janitor batch size
* Update CHANGELOG.md
---------
Co-authored-by: Agung Hariadi Tedja <agung.tedja@kumparan.com>
