Weekly batch stuck #178
-
|
It seems that the weekly batch 79 is stuck at 40 of 1000. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 8 replies
-
|
@zl0v7hzr Thank you for the report! This is really helpful, as the issue is resulting in unnecessary and expensive cloud costs. I have not yet identified the root cause, but it appears that the Docker containers are failing to communicate with the HTTP client, which is preventing new matchings from being created. I am going to restart the VM and then attempt to manually resume the matching process. |
Beta Was this translation helpful? Give feedback.
-
|
I think #176 may be related. If a submission archive takes around 11 to 12 seconds to be extracted by I agree that timeouts for launching containers (and the HTTP servers inside) are necessary, but the matching program should retry or continue with another combination of agents after timeouts are exceeded. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your comment. It's not yet clear whether increasing the timeout parameter from 10 to 20 will resolve the issue. I don't think the problem is caused by I/O load, for the following reasons:
I'll investigate the kernel logs and other relevant information to further understand the root cause. Additionally, to make it easier to isolate the problem, I'll make the following changes today:
I'll provide another update as soon as I have more information on this matter. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
I have another guess about an indirect cause of issues, although I am not sure. In the I am not sure whether you have ensured to delete all containers outside the call to |
Beta Was this translation helpful? Give feedback.
-
|
Okay! I've finally changed my mind. Increasing the timeout from 10 to 20 definitely seems worth it. I will fix it now.
I'm confident that old containers are deleted before the match is resumed. I will comment later with more details, as I have something to do from 20:00 JST. Sorry! Changes regarding the timeoutsI've manually stopped the batch processing now. I will restart the batch processing AFTER changing the following two timeouts:
Patch: da771f9 .. BTW, I saw this log, and experiencing two timeout errors in the same game is incredibly stressful. I sincerely apologize for the very poor experience! Thanks a lot! Update(19:51 JST): The batch has been resumed. Matches after 19:48 JST should have a timeout setting of 20. |
Beta Was this translation helpful? Give feedback.
-
|
@smly I found that my bot much more frequently timed out (resulting in error type events) after you restarted the weekly batch. Were there any background tasks running on the server that used some significant CPU? (such as I believe somehow my bot took more than 3 times the normal time to produce some actions, but the time for my bot to derive each action should be almost the same regardless of the game states, so it must be that the actual CPU time available to my bot was much less than 2 seconds (probably less than 1 second) for those. My bot run without any errors for the last weekly batch, but it suddenly timed out in the same game multiple times, and the timeouts happened close to each other in time. So I suspect that there were some other CPU intensive background tasks running. Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
Hi! Thank you so much for your valuable feedback! I logged into the VM and monitored the processes, but I didn't find any CPU-intensive background tasks running. No anomalies in syslog. However, I found a condition that could cause problems. CPU resource competition with other botsI found a bot during a game that was taking considerable amount of time for each step with a high CPU load. 2025-02-22.15.30.53.movI just noticed that each container seem to be started with the |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your investigation! May I ask when will you switch to the For example: https://mjai.app/games/79/109/0?k=2&s=91 and the other duplication in this match too. I believe using cpuset should be much more stable in the CPU performance and eliminate (or largely reduce) such fluctuation. The timeouts are happening so frequently that the evaluation of my bot is significantly impacted, and it can be unfair to other bots as well (because the penalty points may not be fair to other players with different seats, rankings, and hands). I will re-upload my submission to the validation server after you make the change to ensure that it works with new docker run configurations. Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
Do not worry. I don't casually make changes that disadvantage developers. I am now considering which of the following options. (a). Use the I think option (b) is better and less risky. I can try it out right from the next batch. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you! I think we can try increasing the timeout to 3 seconds and see whether it solves the issue. Ideally more isolation is better, but it is hard to really isolate everything (e.g. the memory and disk cache can be less effective when multiple bots are running simultaneously). So I think it is acceptable to increase the timeout a bit to accommodate such variability. |
Beta Was this translation helpful? Give feedback.
Okay! I've finally changed my mind. Increasing the timeout from 10 to 20 definitely seems worth it. I will fix it now.
I'm still not sure of the root cause, but it's possible that the HDD performance on the VM has degraded.
I'm confident that old containers are deleted before the match is resumed. I will comment later with more details, as I have something to do from 20:00 JST. Sorry!
Changes regarding the timeouts
I've manually stopped the batch processing now. I will restart the batch processing AFTER changing the following two timeouts:
start_kyokuandstart_gamefrom 10 to…