Deployment hangs after importing bindings file

May 6, 2009 at 2:38 PM
This problem occurs only in our Production and QA environments, but not in our nearly identical dev environment (and also not on my local development box).

During the deployment, the process hangs during the "DeployAndStartPorts" target - just after the import:

...
    BTSTask.exe ImportBindings -Source:"C:\Program Files\MyApp\2.0\Deployment\PortBindings.xml" -ApplicationName:MyApp_2.0
    Microsoft (R) BizTalk Application Deployment Utility Version 3.6.1404.0
    Copyright (c) 2006 Microsoft Corporation. All rights reserved.
    
    Information: Importing bindings "C:\Program Files\MyApp\2.0\Deployment\PortBindings.xml" into application "MyApp_2.0" in BizTalk configuration database (server="xxxx", database="yyyyyy")...
    Information: Imported application "MyApp_2.0" bindings.
    
    Command succeeded with 0 errors, 0 warnings.
^CTerminate batch job (Y/N)? N

I logged this as a problem about a year ago and never really resolved it.  (See here: http://biztalkdeployment.codeplex.com/WorkItem/View.aspx?WorkItemId=1154 )
as you can see from the details of that post, we are not alone in this.
We've just gotten used to hitting control-C to get control of the hung process, and select "N" to let it continue. 
The job finishes and all we have to do is start up the app manually in the admin console.  There are never any errors reported in the console, in the generated log, the event log, nor are there any warnings when we start up manually. 
This DOES apparently stop any other tasks following this one to fail.

We also see this behavior when we undeploy - specifically on the "UndeployAppDefinition" step just after the application is removed:

    BTSTask.exe RemoveApp -ApplicationName:MyApp_2.0
    Microsoft (R) BizTalk Application Deployment Utility Version 3.6.1404.0
    Copyright (c) 2006 Microsoft Corporation. All rights reserved.

    Information: Removing application "MyApp_2.0" from BizTalk configuration database (server="xxxxx", database="yyyyy")...
    Information: Removed application "MyApp_2.0".

    Command succeeded with 0 errors, 0 warnings.

^CTerminate batch job (Y/N)? N

We have reviewed and re-reviewed the bindings files and everything looks fine.  We have tried exporting the bindings of a successful deployment in the admin console and using those as the input of the deployment framework, but the hang still occurs.  The only thing I can think of that is some permissions issue somewhere, as our QA system is more restrictive on permissions that my local dev environment is.  I cannot find anything in the security or application event logs either.

Any thoughts?
Coordinator
May 6, 2009 at 3:43 PM
I should rename DeployAndStartPorts since all it does is configure the bindings.  The steps that come after DeployAndStartPorts on a deploy are:

      DeployVDirs;
      DeployVocabAndRules;
      DeployCustomFunctoids;
      DeploySSO;
      DeployBtsNtSvcExeConfig;
      DeployBam;
      StartApplication;
      CustomPostDeployTarget;
      BounceBizTalk

Which of these other deploy options do you have enabled?  When you Ctrl-C and then answer Terminate - No, what comes next?  Does it literally stop at the "Command succeeded with 0 errors, 0 warnings." and sit there indefinitely?  All we do is call btstask.exe, and we've even switched the execution engine from Nant to MSBuild in the meantime (I believe you are using 5.0?).

Thanks,
Tom
Coordinator
May 6, 2009 at 3:47 PM
Do you see any suspicious behavior in the database at this point ?  Aside from malformed port bindings file, I've seen this problem occur if you happen to have a conflicting deployment going on, or someone is doing something fairly intense with the admin console...

thanks -

Scott
May 6, 2009 at 4:22 PM
Scott, short of running a SQL profile I do not see anything odd in the database activity.  This "hang" is consistent and repeatable in the QA and Production environments.  There are other applications running in the same database, but they are definitely not under any significant load. There are no shared resources between those apps and mine.  Those other apps were deployed "the old fashioned way".  Also, there are no others using the admin console at the time of the hang.

Chris
May 6, 2009 at 4:29 PM
Tom, there were no vDirs, Vocab, CustomFunctoids, SSO, or BAM pieces.  The Start and Bounce steps never executed, of course.  I created a dummy app with a VDir and that step does also fail to execute.  So it appears that my stop and continue didn't actually continue anything - but I never noticed the extra steps until recently.

Chris
May 6, 2009 at 4:33 PM
Oh, and yes, if we do nothing, it will sit there indefinitely.  When we do the cancel/resume the logfile shows nothing more, but the command window shows this:

Command succeeded with 0 errors, 0 warnings.

^CTerminate batch job (Y/N)? N
-----
Build started 5/6/2009 8:57:32 AM.
__________________________________________________
Project "C:\Program Files\MyApp\2.0\Deployment\Framework\CopyDeployResults.msbuild" (default targets):

Target Copy:
    Copying file from "..\..\DeployResults\DeployResults.txt" to "..\..\DeployResults\DeployResults_machinename_05062009_0857.txt".

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:00.18
Press any key to continue . . .
May 6, 2009 at 4:40 PM
This is version 5 of the BTDF.  This issue did occur in the older Nant version as well, however
May 6, 2009 at 4:52 PM
I found this
http://geekswithblogs.net/RobBowman/archive/2009/02/17/btstask---import-bindings.aspx
But I am not sure if it is related.  The symptoms are similar, but this would not explain why it works in some environments and not others...
Coordinator
May 6, 2009 at 5:38 PM
Since the hangs are not happening on the same step between deploy and undeploy, and this happened under both Nant and MSBuild, the only common thread is calling BTSTask.exe.  I have to think that there is something going on with BTSTask.exe not returning a result code.  At the point where it is hung up, is there a BTSTask.exe process still running?  If so, what happens to the script if you kill the BTSTask.exe process?
Coordinator
May 6, 2009 at 6:14 PM
(I would indeed run a sql profile to see whether BTSTask.exe has any active SPIDs when it is hanging...)
May 6, 2009 at 7:37 PM
Just a quick note - we've found some blocking in the database.  We've got some DBA's digging in to see if it is related.  I'll post my findings here.
May 7, 2009 at 6:03 PM
Edited May 11, 2009 at 7:47 PM

Blocking was related to something else and has been eliminated.  Same issues still exist.  We're doing more database profiling.  Nothing is "hung" per-se, but I can see that the last line that gets execited in SQL by the BTSTask executable is:

select SSOServerName from adm_Group


Coordinator
May 11, 2009 at 3:38 PM

Did you see my previous reply?  Were you able to take a look at the process list?

Thanks,
Tom

May 11, 2009 at 7:51 PM

On the machine that the deployment is running on, the only new processes that show up as running are btstask.exe and cmd.exe - and both of those drop to zero activity during the hang.  I hadn't noticed, but when I do the control-C thing, it actually leaves both processes present.  During our repeated database tracing test, I found several copies of cmd and btstask in memory.  Their presence doesn't impact the behavior though - I get the same thing from a freshly rebooted machine as well.

Coordinator
May 13, 2009 at 5:16 AM

So that would suggest that btstask.exe is hanging, either because it's crashing, waiting indefinitely or displaying a dialog box.  Did you try killing the hung btstask.exe instead of Ctrl-C to see if the rest of the script finishes normally?

 

May 19, 2009 at 6:49 PM

When I killed the BTSTask process, the script continued with the build failed message as follows:

                Information: Imported application "MyApp_2.0" bindings.

                Command succeeded with 0 errors, 0 warnings.

                C:\Program Files\MyApp\2.0\Deployment\Framework\BizTalkDeploymentFramework.targets(1408,5): error MSB3073: The command "BTSTask.exe ImportBindings -Source:"C:\Program Files\MyApp\2.0\Deployment\PortBindings.xml" -ApplicationName:MyApp_2.0" exited with code 1.
            Done building target "DeployAndStartPorts" in project "MyApp.Deployment.btdfproj" -- FAILED.
        Done building target "DeployBizTalkMgmtDB_true" in project "MyApp.Deployment.btdfproj" -- FAILED.
    Done building target "ServerDeploy" in project "MyApp.Deployment.btdfproj" -- FAILED.
Done building target "Deploy" in project "MyApp.Deployment.btdfproj" -- FAILED.

Done building project "MyApp.Deployment.btdfproj" -- FAILED.

Build FAILED.
C:\Program Files\MyApp\2.0\Deployment\Framework\BizTalkDeploymentFramework.targets(1408,5): error MSB3073: The command "BTSTask.exe ImportBindings -Source:"C:\Program Files\MyApp\2.0\Deployment\PortBindings.xml" -ApplicationName:MyApp_2.0" exited with code 1.
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:02:00.37
-----
Build started 5/19/2009 2:40:25 PM.
__________________________________________________
Project "C:\Program Files\MyApp\2.0\Deployment\Framework\CopyDeployResults.msbuild" (default targets):

Target Copy:
    Copying file from "..\..\DeployResults\DeployResults.txt" to "..\..\DeployResults\DeployResults_machinename_05192009_1440.txt".

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:00.09
Press any key to continue . . .

May 19, 2009 at 7:06 PM

I hadn't noticed, but MSBuild.exe is also an active process at the time of the hang, but I guess that makes sense, as it would be at the top of the call stack, right?

MSBuild -> cmd -> BTSTask

Coordinator
May 27, 2009 at 10:48 PM

That's right, so there is no question that BTSTask.exe is hanging.  MSBuild will wait indefinitely for it to exit and pull the program return code.  You can easily run the same command outside of the script.  Just run to the point that it hangs, then kill BTSTask.exe.  From a Command Prompt, run BTSTask.exe ImportBindings -Source:"C:\Program Files\MyApp\2.0\Deployment\PortBindings.xml" -ApplicationName:MyApp_2.0.  See if you have a similar issue.  It may be something to do with your binding file that is causing BTSTask to hang.