Task queue processor
What is the dataskq processor?
DirectAdmin processes many different tasks, some of them are immediate while others are scheduled to be processed by an internal processor - the task queue manager (
/usr/local/directadmin/dataskq). It is executed by main
directadmin service every minute (can be changed with dataskq_run_interval config option).
Similar tasks are grouped and executed only once. For example, adding new domain, subdomain, enabling SSL requires a reload of the WWW server. If there are multiple tasks queued to reload WWW server, reload would be executed only once. This means that you have to wait at least 1 minute after doing some vague changes to the server before the dataskq processor is run.
The task queue is stored in the
/usr/local/directadmin/data/task.queue file. When queue processing is finished, the file is removed. It is normal to find this file absent on the system.
Tasks can be input into the task.queue file manually to be processed by
dataskq. It is just the syntax that should be known for this. Most common examples:
- restart directadmin:
echo "action=directadmin&value=restart" >> /usr/local/directadmin/data/task.queue
- rewrite user's WWW server configuration:
echo "action=rewrite&value=httpd" >> /usr/local/directadmin/data/task.queue
- update DirectAdmin service:
echo "action=update&value=program" >> /usr/local/directadmin/data/task.queue
- process nightly tally:
echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue
Dataskq could be launched manually without waiting for the cronjob to process it. It also accepts different debug levels:
Where d80 is debug level. Commonly used levels are d80, d400, d800, d2000.
What is the task.queue.cb file
There could be another task.queue file
/usr/local/directadmin/data/task.queue.cb which is always parsed and mostly is filled by the custombuild script. Could be used same way like the 'real'
The "feature" part is that the dataskq can now be called like this:
/usr/local/directadmin/dataskq d2000 --custombuild
With the --custombuild option passed to the dataskq (with or without debug mode), this tells the dataskq to skip all other tasks, and only process the task.queue.cb file.
task.queue.cbfile is executed.
- No other task.queue or task.queue.da file is executed
- Services are not checked if they're running
- Partition usage is not checked
- Brute force scanner is not run
- The DA login history counter is not checked (for port 2222)
Only one instance of the dataskq at a time
Since dataskq is being run every minute if things like backups are issued at different times, say 5 minutes after the first run of a backup, then you'll end up having 2 backups being created at the same time, which can generate slow performance because of the drive being forced to bounce between 2 locations on disk simultaneously.
The solution for that is to only have 1 instance of the dataskq ever running at one time. Note that the consequence to that is that any actions that needs to be run while a backup is executing will not be processed. E.g., service monitoring, service restarts, or other task.queue commands. They will be stored in the
task.queue file and executed once the first instance of the dataskq has finished.
It is possible to limit the number of concurrent
dataskq process with the dataskq_max_instances config parameter. Setting it to
1 will ensure no new
dataskq processes will be started until the already running one will finish its tasks.
da config-set dataskq_max_instances 1 --restart
Note: We do not recoomend changing the default. Despite being slower concurrent
dataskq processes ensures all changes and background tasks are executed in a timely manner. Please use this config parameter only as a last resort. To minimize the potential impact it could be set to
5 instead of
I want the dataskq to run more than once per minute
It can be controlled with the dataskq_run_interval config option. Running
dataskq every 30 seconds:
da config-set dataskq_run_interval 30s --restart
The dataskq is causing 100% CPU
If you're noticing your system load is on the rise, and you can see that the "dataskq" binary is running at the top of the list (in "top"), this guide will help you debug what it's doing.
/var/log/directadmin/system.logto see if the tally is being run. If Users continue to be added to the log (the logs are doing something), then it's likely just the nightly tally, which is normal (assuming the log continues to grow and isn't just stuck on one User).
The first thing to do, is to simply ask the dataskq what it's up to. To do this, type:
killall -USR1 dataskq tail -n 10 /var/log/directadmin/errortaskq.log
What this will do is dump the running dataskq's current process location to the errortaskq.log. What that log output says will determine what to do next.
If the output makes any reference to Maildir along with a path, then what it likely means is that the mentioned path contains an over-sized inbox. Check that inbox and delete the messages, if the email user doesn't seem to be deleting them.
If the output makes any reference to brute_force or some related file, then the cause is likely the dataskq chewing on the system logs with many entries.
- First, ensure you're using the latest version of DirectAdmin.
- Then check:
cd /usr/local/directadmin/data/admin ls -la brute_force*
We're checking for a large
brute_log_entries.list file. If it seems to be "large" (say, over a Meg in size), then that can slow things down. You can safely delete "brute_log_entries.list", as it's not part of the Brute Force Monitor's (BFM's) counting...it's only for your own purposes, showing you what each attack was. DA will recreate this file automatically.
- If the
brute_log_entries.listcontinues to grow and you'd like it to keep itself smaller, go to Admin Level -> Admin Settings and lower the value for Clear failed login attempts from log X days after entry was made. to something around 2 days.
- Also, increasing the values for Notify Admins after an IP has X login failures on any account and Notify Admins after a User has X login failures from any IP. will reduce the number of entries made into the
My monthly bandwidth is not resetting
The monthly bandwidth reset is done at 4:20am on the first day of the month. If you find that this did not happen (allow about an hour for the dataskq to run, pending the number of sites), then there are a few steps you can take to find out why, and reset it manually.
The first thing to check is the cron logs to ensure that the reset was actually issued. Check the
/var/log/cron for the 4:20am timestamp for the 1st day of the month. You should see something like:
echo 'action=reset&value=all' >> /usr/local/directadmin/data/task.queue
If you don't see that command, then you'll want to check this guide to ensure that crond is running and the
/etc/cron.d/directadmin_cron file is working correctly.
If you do see the above command in your cron log, then the next step is to check
/var/log/directadmin/system.log for the entry that looks something like this:
2019:04:01-04:20:01: Reset all command Received
If you don't see that log entry, then you'd again want to use the same guide as above to debug the issue.
If you do see the above log entry, then scroll just below it to confirm that each user is mentioned. If not, then likely the users are not all set in the
users.list file. You can use this guide to reset all users.list files.
At this stage, hopefully you would have figured out why the reset didn't happen, and have fixed it so it does reset all users' bandwidth next time. However, the reset still wouldn't have happened for the past reset. So to do that, you'd use the cleanreset task.queue command. Type:
echo "action=cleanreset&value=all" >> /usr/local/directadmin/data/task.queue /usr/local/directadmin/dataskq d echo "action=tally&value=all" >> /usr/local/directadmin/data/task.queue
which should reset things from the start of the month without losing data. Manually doing the regular reset mid-month will lose current data that you still want, hence we use the cleanreset, which scans all datestamps to ensure it doesn't delete anything it shouldn't.
If you didn't figure out why the reset didn't happen, you may also want to check for other competing tasks. For example, if the server is rebooted at the time of the reset, that could prevent it from working. Also, the dataskq is also used by other tasks such as backups, so check to see if you've got any large backups running near the same time as the reset, and check
/var/log/directadmin/errortaskq.log for any errors relating to them.
Make the bandwidth update more often
Cron is the tool used to control the action of the stats tally. It is performed once 10 minutes after midnight every day.
If you want to run it more often a new cron script can be created with additional requests to perform tally. Example of
10 6 * * * root echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue 10 12 * * * root echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue 10 18 * * * root echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue
Would execute tally at 00:10, 06:10, 12:10 and 18:10. Note that 00:10 is default tally execution time, additional times are specified in new file.
One important thing to remember is that for very large servers, with upwards of 1000 domains, the tally can take a significantly longer time to run (up to 2 hours), so if you run a large server, you may not want to run the tally too often.
Once you've set the value you want, save the file and reload the cron daemon:
service crond restart
WWW server graceful restarts after the nightly tally
After the nightly tally runs, the dataskq will issue a graceful restart to the WWW server to both re-open logs and reload configs in case accounts have been suspended/unsuspend:
If you wish to disable this (not recommended), you may use this:
tally_post.sh to take some special action that you want (note that it's called just before the graceful restart, as the restart is sent to the task.queue to be processed in the next minute/dataskq run).
How to delete domains from the command line
Domain deletion could be passed to the dataskq process with this command:
echo "action=delete&value=domain&requestedby=admin&select0=domain.com(&select1=domain2.com...)" >> /usr/local/directadmin/data/task.queue
Where options are:
- requestedby=admin : controls where the result Message is sent to within the Message System, where the result is either success or fail. It does not have to be the owner of the domain, but can be. The actual domain owner is looked up for each domain.
- select0=domain.com : the domain to be deleted.
- select1=domain2.com : optional other domains, for any User, does not need to be under the same User. (select2=, select3=, etc.)
How to link IPs trough the task.queue
Related to interface Admin Level -> IP Manager -> click on an IP -> Link IP, this can now be done through the task.queue.
echo "action=linked_ips&ip_action=add&ip=188.8.131.52&ip_to_link=184.108.40.206&apache=yes&dns=yes&apply=yes" >> /usr/local/directadmin/data/task.queue
The command is the same as the form in CMD_IP_MANAGER_DETAILS?ip=220.127.116.11, except we move the form action=add to ip_action=add, as the dataskq action needs to be broader.
How to Message Users From the Command Line
To message a specific account, list of accounts or all accounts (Admins+Resellers+Users), the task.queue could be used. Format is as follows:
action=notify value=users users=all|select1=fred&select2=george&select3=gary subject=Your Subject message=Content of your Message
where the "users" variable can be "all" (meaning DA will message all accounts on the server) or can be a list of accounts that are url-encoded with select1, select2.
For a single account, just use:
Note that the = and & characters in the users list must be url encoded with hex (
%26, respectively). Also, if you want to include a newline in the message, use
A sample echo command is:
echo 'action=notify&value=users&subject=Hello World&message=Hello,%0A%0A This is a message.&users=select1%3Dadmin%26select2%3Dgary' >> /usr/local/directadmin/data/task.queue
How to send notification to the Admin Message System
This is a very simple new task.queue command to send Admin notices.
action=notify value=admin subject=Your Subject message=Content of your Message
It should be URL encoded (mainly if you want newline characters:
%0A) so that it can be posted to the task.queue.
Here's an example:
echo "action=notify&value=admin&subject=hello this is my subject&message=this is the notification%0Awith a new line." >> data/task.queue.cb; ./dataskq d10 --custombuild
Note: the above sends to the
task.queue.cb file, and calls the dataskq with the
This skips all other typical dataskq commands, so you don't accidentally process a backup or the tally.
How to run bandwidth only tally
You may force dataskq to run bandwidth only tally using the following command:
echo "action=bandwidthtally&value=all" >> /usr/local/directadmin/data/task.queue
Alternatively to process per-user:
echo "action=bandwidthtally&value=all&value=user&type=username" >> /usr/local/directadmin/data/task.queue
How to run a quota only tally
When you delete a domain, the disk usage and totals are recounted for the domain as the deletion actually runs a 'mini' tally for quota only, like:
echo "action=quotatally&value=bob&type=user" >> /usr/local/directadmin/data/task.queue
You may recalculate the 'quotatally' for all users like so:
echo "action=quotatally&value=all" >> /usr/local/directadmin/data/task.queue
system.log, the usual tally entries will show up, but with this, it will show the mask, e.g.,
Tally User admin Begin: mask 20
This lets you know what the tally is actually looking at. If no mask is present, e.g.,
Tally User admin Begin
then everything is checked.
The mask layout is as follows:
#define TLY_MASK_WEBALIZER 1 #define TLY_MASK_BANDWIDTH 2 #define TLY_MASK_DISKUSAGE 4 #define TLY_MASK_LOGROTATE 8 #define TLY_MASK_TOTALS 16 #define TLY_MASK_SUSPEND 32
so, that means a mask of 20 = 16+4, so we have totals + quotas in the above example.
How to set quota limits for users
This command will go through each user, and set the quota limits in the system quotas. Users with unlimited quotas will also have their quota limit set to 0 (unlimited):
echo "action=rewrite&value=quota" >> /usr/local/directadmin/data/task.queue
Or per user:
echo "action=rewrite&value=quota&user=bob" >> /usr/local/directadmin/data/task.queue
How to rewrite a DNS zone
The domain key could be used to rewrite a DNS zone for only that domain:
echo "action=rewrite&value=named&domain=domain.com" >> /usr/local/directadmin/data/task.queue
Also, an optional key could be set to not update the DNS zone's serial number:
If update_serial is not passed, yes is assumed.
echo "action=rewrite&value=named&domain=domain.com&update_serial=no" >> /usr/local/directadmin/data/task.queue
How to recreate spambox settings for mailboxes
When you make change in your SpamAssassin settings, each time you click "Save", DA will recreate/set the ownership/permissions on each spambox file.
If you have a large number of pop accounts, this can take long enough that the timeout is reached. Moving this task to the task.queue file to be run in the background will allow things to run quicker in the foreground in DA.
The actual task.queue command:
echo "action=defaultspam&domain=domain.com&username=user" >> /usr/local/directadmin/data/task.queue
A command to rewrite spam filters is:
echo "action=rewrite&value=filter" >> /usr/local/directadmin/data/task.queue
How to reset monthly stats
In the case that the monthly reset did not run, an admin can run:
echo "action=cleanreset&value=all" >> /usr/local/directadmin/data/task.queue
in order to only remove the previous month's data.
This will do a standard reset of
bandwidth.tally files and
user.usage files, but will not follow with a tally.
You'd need to run the full tally command afterwards to recount the correct stats for Reseller Level and Admin Level counting:
echo "action=tally&value=all" >> /usr/local/directadmin/data/task.queue
How to rewrite virtual user files
To rewrite/convert the
/etc/virtual/domain.com/passwd files, you may use following command:
echo "action=rewrite&value=email_passwd&user=fred" >> /usr/local/directadmin/data/task.queue
Where fred is desired user that owns domain.com.
Or, to rewrite all, run:
echo "action=rewrite&value=email_passwd" >> /usr/local/directadmin/data/task.queue
How to rewrite apache/nginx configs for single user
Instead of rewriting configs for all users, save time by rewriting a single account's config as follows:
echo "action=rewrite&value=httpd&user=fred" >> /usr/local/directadmin/data/task.queue
List of other taskq commands
A list of other useful taskq commands that one could echo into the
task.queue file are outlined below.
Regenerate vacation config files, first for all, and then for a single domain:
Regenerate email filters, first for all, and then for a single user:
Regenerate email passwd files, first for all, and then for a single user:
Regenerate email aliases, first for all, and then for a single user:
helo_data, first for all, and then for a single domain/ip:
action=rewrite&value=domainips action=rewrite&value=domainips&domain=domain.com action=rewrite&value=helo_data action=rewrite&value=helo_data&ip=18.104.22.168
Run sync to cluster with 10 retries (in case one server is unreachable):
Run sync to cluster email data for the user fred to host 22.214.171.124 (failed host), with 10 retries and the longrequest URL encoded POST from the original client request:
Run domain verification process of
Check if user/file exists and ensure the lines in Apache configs are correct based on that:
Regenerate the users' pureftpd database:
Regenerate dkim 1) for all domains, 2) for a single domain, and 3) for a single domain and add the key to the domain's DNS:
action=rewrite&value=dkim action=rewrite&value=dkim&domain=domain.com action=rewrite&value=dkim&domain=domain.com&dns=yes
Forcefully run BFM to scan Apache logs 1) for all domains, 2) for a list of users, and 3) for a list of domains:
action=rewrite&value=brute_force_scan_apache_logs action=rewrite&value=brute_force_scan_apache_logs&users=...userslist... action=rewrite&value=brute_force_scan_apache_logs&domains=...domainslist...
Update/check the panel license:
Update the DirectAdmin panel itself:
action=backup&type=user&value=john action=backup&owner=reseller&type=reseller&value=multiple&when=now&where=local&who=all action=backup&type=reseller&value=multiple&owner=bob&where=local&when=now&select0=george&select2=eddy.. [ftp_ip=126.96.36.199&ftp_username=asdf&ftp_password=base64enc&ftp_path=/] action=backup&type=reseller&value=multiple&owner=bob&where=local&when=now&select0=buger action=backup&value=now&type=system action=backup&type=sitebackup&value=bob&password=pass&select0=ftp&select1=email, etc (backup options)
Notify user 'bob' about blacklisted IP '188.8.131.52':
Regenerate the caches used throughout the DirectAdmin panel:
action=cache&type=popquota action=cache&type=popquota&value=domain.com action=cache&value=showallusers action=cache&value=showallusers&user=bob action=cache&value=showallusers&select0=bob&select1=fred&... action=cache&value=safemode action=cache&value=safemode&domain.com=username&domain2.com=user2
Add or delete DNS records:
action=dns&do=add&domain=es5.com&type=A&name=taskqrec&value=184.108.40.206&ttl=500 action=dns&do=delete&domain=domain.com&type=A&name=some.thing.&value=220.127.116.11 action=dns&do=delete&domain=domain.com&type=A&name=some.thing.&value=* //deletes any value. action=dns&do=delete_zone&domain=domain.com
Delete old mbox files (nowadays Maildir format is in use):
Convert mailboxes to use new mail partitioning:
action=convert&value=mail_partition action=convert&value=mail_partition&smart=yes action=convert&value=mail_partition&user=fred action=convert&value=mail_partition&user=fred&smart=yes action=delete&value=old_mail_partition action=delete&value=old_mail_partition&user=fred
Delete users, delete domains, and lastly, delete databases:
action=delete&value=account&requestedby=admin&select0=bob(&select1=fred...) action=delete&value=domain&requestedby=admin&select0=domain.com(&select1=domain2.com...) action=delete&value=test_dbs
Switch from one IP address to another:
Update plugin (CustomBuild GUI plugin example):