ERROR: Exception caught in WorkDoerFactory #134

ssri28 · 2019-11-04T09:15:52Z

Hi Team,

I am getting error in running Canvas germline caller.

ERROR:
ERROR: Exception caught in WorkDoerFactory. Cancelling all jobs. Exception:
One or more errors occurred. (One or more errors occurred. (Job tabix-index-tmp-ballele.bedgraph.gz.1 failed with exit code 139))

cmd:
/home/ec2-user/dotnet/dotnet /home/ec2-user/data/Canvas-1.40.0.1613+master_x64/Canvas.dll SmallPedigree-WGS -b L2.sorted.rmdup.bam --sample-b-allele-vcf L2.HC.vcf -n MSC --ploidy-vcf canvas_ploidy.vcf -o MSC_cnv -r /home/ec2-user/data/genome/hg19/hg38.fa -g /home/ec2-user/data/genome/hg19/ -f exclude_intervals.bed

Kindly help me to fix the bug.

thanks a lot

eroller · 2019-11-04T18:11:12Z

run chmod +x on the tabix executable

ssri28 · 2019-11-05T06:09:07Z

Hi,
Thanks for the reply.
chmod +x on the tabix executable did not work.

ssri28 · 2019-11-05T16:54:33Z

Hi, Thanks for the reply. chmod +x on the tabix executable did not work. Still i am getting same error

…

On Mon, 4 Nov 2019 at 23:41, Eric Roller ***@***.***> wrote: run chmod +x on the tabix executable — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#134?email_source=notifications&email_token=AKXA3TTSOCTQZSVZSAB654TQSBQUDA5CNFSM4JIQYGN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDAGBVI#issuecomment-549478613>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKXA3TQ3T6KWHFJVZO7UMNLQSBQUDANCNFSM4JIQYGNQ> .

eroller · 2019-11-05T17:00:56Z

The std out should show the command that is failing. Can you make sure this is the same tabix executable you updated permissions on. There may be more than one tabix executable. You can also just try: chmod -R 777 /home/ec2-user/data/Canvas-1.40.0.1613+master_x64

ssri28 · 2019-11-06T01:31:55Z

Hi eroller,

I even tried it..

Error i am getting

ERROR: Exception caught in WorkDoerFactory. Cancelling all jobs. Exception:
Type specified in JSON 'CanvasCommon.CommandLineParsing.OptionProcessing.SuccessfulParsingResult1[[Canvas.CommandLineParsing.SmallPedigreeInput, Canvas, Version=1.40.0.1613, Culture=neutral, PublicKeyToken=null]], CanvasCommon, Version=1.40.0.1613, Culture=neutral, PublicKeyToken=null' is not compatible with 'CanvasCommon.CommandLineParsing.OptionProcessing.IParsingResult1[[Canvas.CommandLineParsing.GermlineWgsInput, Canvas, Version=1.40.0.1613, Culture=neutral, PublicKeyToken=null]], CanvasCommon, Version=1.40.0.1613, Culture=neutral, PublicKeyToken=null'. Path '$type', line 2, position 157.
Newtonsoft.Json.JsonSerializationException: Type specified in JSON 'CanvasCommon.CommandLineParsing.OptionProcessing.SuccessfulParsingResult1[[Canvas.CommandLineParsing.SmallPedigreeInput, Canvas, Version=1.40.0.1613, Culture=neutral, PublicKeyToken=null]], CanvasCommon, Version=1.40.0.1613, Culture=neutral, PublicKeyToken=null' is not compatible with 'CanvasCommon.CommandLineParsing.OptionProcessing.IParsingResult1[[Canvas.CommandLineParsing.GermlineWgsInput, Canvas, Version=1.40.0.1613, Culture=neutral, PublicKeyToken=null]], CanvasCommon, Version=1.40.0.1613, Culture=neutral, PublicKeyToken=null'. Path '$type', line 2, position 157.
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolveTypeName(JsonReader reader, Type& objectType, JsonContract& contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, String qualifiedTypeName)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadMetadataProperties(JsonReader reader, Type& objectType, JsonContract& contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue, Object& newValue, String& id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
at Isas.Framework.Checkpointing.Internals.CheckpointJsonSerializer.Deserialize[T](IFileLocation path)
at Isas.Framework.Checkpointing.Internals.CheckpointJsonSerializer.Load[T](String checkpoint)
at Isas.Framework.Checkpointing.Internals.CoreCheckpointRunner.RunCheckpoint[TResult](String name, Func2 run) at Canvas.CommandLineParsing.ModeParser1.Parse(MainParser main, ILogger logger, ISettings settings, ICheckpointRunner checkpointRunner, IWorkDoer workDoer, String[] args, TextWriter standardWriter, TextWriter errorWriter)
at Canvas.CommandLineParsing.MainParser.<>c__DisplayClass20_2.b__2(ICheckpointRunner checkpointer)
at Isas.Framework.Checkpointing.CheckpointerFactory.<>c__DisplayClass1_1.b__1(CheckpointManager checkpointManager)
at Isas.Framework.Utilities.IArgumentSupplierExtensions.<>c__DisplayClass0_02.<SupplyArgument>b__1(TFill arg) at Isas.Framework.Checkpointing.Internals.CheckpointManagerFactory.<>c__DisplayClass4_01.b__0()
at Isas.Framework.Utilities.IArgumentSupplierExtensions.<>c__DisplayClass0_02.<SupplyArgument>b__0() at Isas.Framework.Utilities.IArgumentSupplierExtensions.<>c__DisplayClass0_02.b__1(TFill arg)
at Isas.Framework.Utilities.TaskTrackerFactory.<>c__DisplayClass2_01.<SupplyArgument>b__0() at Isas.Framework.Utilities.IArgumentSupplierExtensions.<>c__DisplayClass0_02.b__0()
at Isas.Framework.WorkManagement.JobLaunching.JobLauncherFactory.RunWithJobLauncher(ILogger logger, ISettings settings, IDirectoryLocation loggingDir, Action1 logCommand, CancellationToken cancellationToken, Action1 function)
at Isas.Framework.WorkManagement.JobLaunching.JobLauncherFactory.RunWithJobLauncher(ILogger logger, ISettings settings, IDirectoryLocation analysisFolder, CancellationToken cancellationToken, Action1 function) at Isas.Framework.WorkManagement.ResourceManagement.WorkResourceManagerFactory.RunWithResourceManager(ILogger logger, ISettings settings, CancellationToken cancellationToken, Action1 function)
at Isas.Framework.WorkManagement.WorkDoerFactory.RunWithWorkDoer(ILogger logger, ISettings settings, IDirectoryLocation analysisFolder, CancellationTokenSource cancellationTokenSource, Action1 function) at Canvas.CommandLineParsing.MainParser.<>c__DisplayClass20_0.<Run>b__0(ILogger logger) at Isas.Framework.FrameworkFactory.IsasFrameworkFactory.RunWithLogger(IFileLocation workflowLog, IFileLocation workflowError, Action1 function)
at Canvas.CommandLineParsing.MainParser.Run(String[] args, TextWriter standardOutput, TextWriter standardError)
at Canvas.Program.Main(String[] args)

eroller · 2019-11-06T17:06:06Z

Try again using an empty output directory

ssri28 · 2019-11-06T22:20:05Z

Hi Eroller,

Thank for the reply..

i tried giving empty folder.

Again i am getting errors:
Detailed Error Report :
2019-11-06T22:15:43+00:00,ERROR: Job CNV.vcf.gz failed with exit code 134. Job logs:
/home/ec2-user/data/human_exome/ES_MSC_cnv_t/Logging/CNV.vcf.gz.stdout
/home/ec2-user/data/human_exome/ES_MSC_cnv_t/Logging/CNV.vcf.gz.stderr
Job error message:
Unhandled Exception: System.InvalidOperationException: Sequence contains no elements
at System.Linq.Enumerable.Average(IEnumerable1 source) at CanvasDiploidCaller.CanvasDiploidCaller.CallVariants(String variantFrequencyFile, String inFile, String outFile, String ploidyVcfPath, String referenceFolder, String sampleName, String truthDataPath) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasDiploidCaller\CanvasDiploidCaller.cs:line 298 at CanvasDiploidCaller.Program.Main(String[] args) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasDiploidCaller\Program.cs:line 98 2019-11-06T22:15:43+00:00,ERROR: Canvas workflow error: Isas.Framework.WorkManagement.JobFailedException: Job CNV.vcf.gz failed with exit code 134 at Isas.Framework.WorkManagement.JobLaunching.BasicJobLauncher.<RunSystemProcessCheckExitAsync>d__7.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Isas.Framework.WorkManagement.JobLaunching.RuntimeTrackingJobLauncher.<LaunchJobAsync>d__6.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Isas.Framework.WorkManagement.IJobLauncherExtensions.LaunchJob(IJobLauncher jobLauncher, JobInfo jobInfo, Action1 stdoutCallback, Action1 stderrCallback) at Isas.Framework.WorkManagement.IWorkDoerJobExtensions.<>c__DisplayClass0_01.b__0(WorkResources sr, IJobLauncher jobLauncher)
at Isas.Framework.WorkManagement.ResourceManagement.ThreadedWorkResourceManager.<>c__DisplayClass11_01.<RunWithResources>b__3(WorkResources resources) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Isas.Framework.WorkManagement.ResourceManagement.ThreadedWorkResourceManager.<<RunWithResources>b__11_1>d1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Isas.Framework.WorkManagement.AsTaskExtensions.Await[T](Task1 task) at Canvas.CanvasRunner.RunGermlineCalling(IFileLocation partitionedPath, CanvasCallset callset, String ploidyVcfPath, IFileLocation canvasSnvPath) at Isas.Framework.Checkpointing.Legacy.LegacyCheckpointRunner.<>c__DisplayClass0_0.<RunCheckpoint>b__0() at Isas.Framework.Checkpointing.Internals.SerializingCheckpointRunner.<>c__DisplayClass8_01.b__0(Checkpoint checkpoint)
at Isas.Framework.Checkpointing.Internals.CoreCheckpointRunner.RunCheckpoint[TResult](String name, Func`2 run)
at Isas.Framework.Checkpointing.Legacy.LegacyCheckpointRunner.RunCheckpoint(ICheckpointRunner runner, String name, Action a)
at Canvas.CanvasRunner.d__38.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Canvas.CanvasRunner.CallSample(CanvasCallset callset)
at Canvas.ModeLauncher.Launch()

eroller · 2019-11-06T22:41:38Z

So far I have seen three different error messages, with only one command line indicating you are running SmallPedigree-WGS workflow. However, the most recent error indicates you are running Germline-WGS? You should be running SmallPedigree-WGS as Germline-WGS has been deprecated.

Please attach these files in the future: CanvasLog.txt, CanvasError.txt and any log files mentioned in CanvasError.txt, for example looking at the above error you should also include: /home/ec2-user/data/human_exome/ES_MSC_cnv_t/Logging/CNV.vcf.gz.stdout
/home/ec2-user/data/human_exome/ES_MSC_cnv_t/Logging/CNV.vcf.gz.stderr

ssri28 · 2019-11-06T22:52:16Z

Hi Eroller,

Thanks for the rapid response..

i am running now SmallPedigree-WGS

I am attaching the error log..
kindly check it
Canvas_error.zip

ssri28 · 2019-11-06T23:54:53Z

Hi Eroller, Thanks for the rapid response.. i am running now SmallPedigree-WGS I am attaching the error log.. kindly check it

…

On Thu, 7 Nov 2019 at 04:11, Eric Roller ***@***.***> wrote: So far I have seen three different error messages, with only one command line indicating you are running SmallPedigree-WGS workflow. However, the most recent error indicates you are running Germline-WGS? You should be running SmallPedigree-WGS as Germline-WGS has been deprecated. Please attach these files in the future: CanvasLog.txt, CanvasError.txt and any log files mentioned in CanvasError.txt, for example looking at the above error you should also include: /home/ec2-user/data/human_exome/ES_MSC_cnv_t/Logging/CNV.vcf.gz.stdout /home/ec2-user/data/human_exome/ES_MSC_cnv_t/Logging/CNV.vcf.gz.stderr — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#134?email_source=notifications&email_token=AKXA3TWE3MEJ4UFXXJJZCXTQSNB2JA5CNFSM4JIQYGN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDIHZSI#issuecomment-550534345>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKXA3TS3R3JCGCK5L52M3GLQSNB2JANCNFSM4JIQYGNQ> .

eroller · 2019-11-07T00:12:59Z

Failed to open file /home/ec2-user/data/human_exome/ES_MSC_cnv_2/TempCNV_/ballele.bedgraph.gz

Can you open that file? What are the contents? Is it a valid gz file?

ssri28 · 2019-11-07T00:29:40Z

Hi Eroller,

Its a empty file.

eroller · 2019-11-07T16:51:28Z

Can you send me the Logging directory inside the output folder?

ssri28 · 2019-11-07T20:24:17Z

Hi Eroller,
i am sending you the logging directory files
Logging.zip

eroller · 2019-11-07T20:35:05Z

It is empty? I don't see any files in there.

ssri28 · 2019-11-07T22:24:27Z

Hi Eroller,

Please check this attachment

Thanks
Logging.zip

eroller · 2019-11-07T22:52:50Z

It looks like all the b-allele VCF entries are being skipped. Sites must have PASS filter with GT field and GQ. Can you show me some lines of the VCF?

ssri28 · 2019-11-07T22:57:29Z

Hi Eroller,
I am attaching few lines of the vcf file
ES_MSC_HM2FCDSXX_L2.HC_sample.vcf.zip

eroller · 2019-11-07T22:58:22Z

yes, try adding PASS to the FILTER column

ssri28 · 2019-11-07T23:01:24Z

Hi Eroller, thanks for the reply, shall i add "PASS" to all Filter columns in the VCF file?

eroller · 2019-11-07T23:25:49Z

typically the FILTER is applied based on the quality of the variant calls (e.g. by using a threshold on GQ). You should consult the documentation for the small variant caller that generated the VCF on how to apply filters.

ssri28 · 2019-11-08T00:31:25Z

Hi Eroller,
Thank you for your suggestion..
now it is working for me.

eroller closed this as completed Nov 4, 2019

eroller reopened this Nov 6, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ERROR: Exception caught in WorkDoerFactory #134

ERROR: Exception caught in WorkDoerFactory #134

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ERROR: Exception caught in WorkDoerFactory #134

ERROR: Exception caught in WorkDoerFactory #134

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!