ElementTunnel and Byte Order Mark

Topics: Bindings File, Settings Management and SSO
Jul 20, 2010 at 12:38 AM

I'm taking a renewed look at using ElementTunnel.exe.  Up to now, on this project, we have not used it and everything is working fine, but we are now parameterizing new items in SQL ports.

I'm getting the same error as reported here: http://biztalkdeployment.codeplex.com/Thread/View.aspx?ThreadId=155327

Looks like it was fixed in 5.0.12.  So I have a really basic question, how do I know which version of BTDF that I am running?  (Add remove programs simply says 5.0).  Is there a releaseNotes.txt or readme. file that would show the version? 

I'm pretty sure we are 5.0.17 or higher, so maybe the bug has returned?

I'm running it like this:

1) I copied my PortBindingsMaster.xml to PortBindingsMasterOrig.xml

2) I ran this:

elementtunnel /i portbindingsmasterorig.xml /x adapterXPaths.txt /o portbindingsmaster.xml /decode

3) Then I tried a local deploy in Visual Studio 2005

Thanks,

Neal Walters

 

Coordinator
Jul 20, 2010 at 4:09 AM

ElementTunnel.exe rarely changes.  I have not made any changes to it since that rework when the bug fix was put in place.  You might check out the encoding of the master binding files included in the samples -- I know they work fine with the included ElementTunnel.exe -- just for comparison's sake.

I posted an answer to the version number question here.

Thanks,
Tom

Jul 20, 2010 at 4:27 PM

 

At first, I had the same problem seems to be happening there; I ran this:

c:\Program Files\Deployment Framework for BizTalk\5.0\Samples\BizTalk2009\BasicMasterBindings\BasicMasterBindings.Deployment>elementtunnel /i portbindingsmaster.xml /x adapterXPaths.txt /o

elementTunnelBinding.xml /encode

Immediately after, I open the output file in VS2005 and see the "chinese" characters.

 

Then I searched for various version of ElementTunnel.exe.  The one in "c:\Program Files\Deployment Framework for BizTalk\5.0\Framework\DeployTools\ElementTunnel.exe" seems to work correctly.

But oddly enough, the one I was using (that was causing the problem) was from doing my own compile from the ElementTunnel source.   Perhaps the corrected source wasn't included in the 5.17 release?

Neal

 

 

Coordinator
Jul 20, 2010 at 4:42 PM
I'm not sure what to say. I checked the code included in the current release and it's correct, and the code was last changed in March. Any reason why you have another copy of ElementTunnel.exe floating around? You should let the Deployment Framework manage that itself unless you are customizing its behavior for some reason. Thanks, Tom
Jul 21, 2010 at 12:31 AM

When I went to look for the .EXE, I just assumed the one from the build would work fine.  It's actually a different size than the other one.

I brought it up in VS2008 to take a look at what it was really doing, just to have a better understanding. 

I guess the best practice is to build an unwrap.cmd file and place in each Deployment directory?  I think that's what we did when Baliji helped set up a couple of years ago.

Thanks,
Neal

 

 

Coordinator
Jul 21, 2010 at 4:07 AM

An unwrap.cmd that does the ElementTunnel.exe /decode on the master binding file?  Yeah, you could do that.  The thing is that it's a one-time operation.  After you do it once you're basically manually maintaining the binding file from there on.  I could add a VS Tools menu option for the decode function...

Thanks,
Tom

Jul 21, 2010 at 5:07 PM

Yes, adding elementTunnel to the ToolBar would be kind of cool (one less thing to chase down).

It seems like during developing, we often make some major changes, then have to decide if it easier to rebuild the PortBindingsMaster.xml from scratch vs. copy/paste and do surgery to put in newly added ports.

I have another basic question, actually our team leader here asked.  Is there really any reason to have the "<!-- else -->" statements in the binding file.  I know the theory is that the binding file would be useable "as is" if needed, but I'm not sure it is ever really needed.  If we always do BTDF deploys, then I would think it is not needed.  Even local deploys use the XmlPreprocess (at least the way we have things configured). 

Neal

 

Coordinator
Jul 21, 2010 at 8:12 PM

Correct, it is probably rare that the else is actually used, presuming that you are always deploying through the Deployment Framework.  It is possible to disable XmlPreprocess, but then you would probably not be using those special comments anyway.  If you don't have them in there then the master binding file is technically invalid -- it probably wouldn't pass an XSD validation -- but in most cases that doesn't matter.

Thanks,
Tom

Jul 23, 2010 at 3:23 PM
Edited Jul 23, 2010 at 4:12 PM

So we can omit the IF-DEFs in the binding files, but still run XMLPreprocess? Or I would just have to wrap the entire binding file in one big "ifdef" statement (one line 1 to the last line)?  

Life with BTDF would get much easier and more liberating when you do a lot of changing and refactoring. 

What about this idea... Why not have a variation (or a sort of reverse) of XMLPreprocess that takes a binding file imported from BizTalk Admin console, and have it change the "Local Development" settings back to the variables?

So for example, if I had a variable called ${MyDatabaseServer}, then it could replace "(local)" (from Column C of the spreadsheet) with ${MyDatabaseServer} (from Column A of the spreadsheet).  This wouldn't cover 100% of the cases (for example, what if I had two access two different SQL servers in prod, but not on dev, but I'll bet it would get 95% or more for most users.

Thanks,
Neal

P.S.  I just tested, no IF-DEFs in the port-binding master did not work (maybe there is a parm to XMLPreprocess that will force it to substitute when no IFDEF's found?). But putting one global IFDEF at the top of the file is working so far.

 

 

Coordinator
Jul 23, 2010 at 4:26 PM
nealwalters wrote:

So we can omit the IF-DEFs in the binding files, but still run XMLPreprocess? Or I would just have to wrap the entire binding file in one big "ifdef" statement (one line 1 to the last line)?  

P.S.  I just tested, no IF-DEFs in the port-binding master did not work (maybe there is a parm to XMLPreprocess that will force it to substitute when no IFDEF's found?). But putting one global IFDEF at the top of the file is working so far.

Sorry, I guess I'm not following what you are trying to accomplish.  There wouldn't be a reason to run XMLPreprocess if you aren't substituting any variables, so why would you leave out the ifdef's?  You can leave out the else's, because XMLPreprocess will always run anyhow and the else's will never be used.

XMLPreprocess can do a lot of stuff, most of which I don't show in the samples.  Loren has added lots of other features to it, which you are free to use in any of your XML files (in addition to PortBindingsMaster.xml, you can XMLPreprocess any XML file by listing it in a FilesToXmlPreprocess ItemGroup).  Check out xmlpreprocess.codeplex.com for all of the syntax and features.

Regarding the variable substitution app, I suppose that could be made to work for most cases, but it would take some effort to get it right.  Probably not a 5.0 feature.

Thanks,
Tom

Jul 23, 2010 at 5:12 PM
Edited Jul 23, 2010 at 5:27 PM

My goal is to reduce human errors with binding files, and to make making changes (adding new ports for example) more flexible. I also want to reduce building the binding files from 30-60 minutes (with errors), down to 5-10 minutes with no errors.

By doing things the hardway, I made several typos that caused deploys to be wacky (see issue under 1 below).  The other goal would be to soften the difficulty of BTDF for newbies. On my new client, I've been indoctrinating and cross-training a co-worker, and trying to keep him from being overwhelmed. 

Let me try to illustrate with a SQL port. 

1. Hardest and stupid way - what I was actually doing last week (not using ElementTunnel) 

        <!-- ifdef ${_xml_preprocess} -->
        <!--     <Address>SQL://${MyDatabaseSQLServer}/${MyDatabaseDatabase}/</Address> -->
        <!-- endif -->

        <TransportType Name="SQL" Capabilities="81163" ConfigurationClsid="6e688659-1374-4f37-90dc-40cb6dd3c73d" />


        <!-- ifdef ${_xml_preprocess} -->
            <TransportTypeData>&lt;CustomProps&gt;&lt;AdapterConfig vt="8"&gt;&amp;lt;Send xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"&amp;gt;&amp;lt;connectionString&amp;gt;Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=${MyDatabaseDatabase};Data Source=${MyDatabaseSQLServer}&amp;lt;/connectionString&amp;gt;&amp;lt;outputRootElementName&amp;gt;SQLInsertCommonErrorResp&amp;lt;/MyRootElement&amp;gt;&amp;lt;documentTargetNamespace&amp;gt;http://MyNameSpace&amp;lt;/documentTargetNamespace&amp;gt;&amp;lt;uri&amp;gt;SQL://(local)/MyDatabase/&amp;lt;/uri&amp;gt;&amp;lt;/Send&amp;gt;&lt;/AdapterConfig&gt;&lt;/CustomProps&gt;</TransportTypeData>
        <!-- else -->

            <TransportTypeData>&lt;CustomProps&gt;&lt;AdapterConfig vt="8"&gt;&amp;lt;Send xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"&amp;gt;&amp;lt;connectionString&amp;gt;Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=MyDatabase;Data Source=(local)&amp;lt;/connectionString&amp;gt;&amp;lt;outputRootElementName&amp;gt;SQLInsertCommonErrorResp&amp;lt;/MyRootElement&amp;gt;&amp;lt;documentTargetNamespace&amp;gt;http://MyNameSpace&amp;lt;/documentTargetNamespace&amp;gt;&amp;lt;uri&amp;gt;SQL://(local)/MyDatabase/&amp;lt;/uri&amp;gt;&amp;lt;/Send&amp;gt;&lt;/AdapterConfig&gt;&lt;/CustomProps&gt;</TransportTypeData>
        <!-- endif -->        

Issues with this pattern:

  1) Before I was using ElementTunnel, this technique was nastier and even more element prone.  But that was my fault for not getting ElementTunnel done correctly.  Errors I would get for example would be to migrate to another system, and then miss one letter in the target namespace (which then doesn't get caught until you run a test), or I would copy patterns that I thought were correct, then forget to change something (for example copying a SendPort file pattern to a ReceivePort file pattern). This was torture and I should have just been shot.  I apparently had some odd issue with ElementTunnel at a prior client with WCF and never overcame the issue at that time.  Sometimes deadlines require the "get-r-done" philosophy.  At that client, I think I spent 2-3 weeks just mastering BTDF, and I don't think most people have that much time or patience.

 

2. Use of Element Tunnel

        <!-- ifdef ${_xml_preprocess} -->
        <!--     <Address>SQL://${MyDatabaseSQLServer}/${MyDatabaseDatabase}/</Address> -->
        <!-- else -->
        <Address>SQL://(local)/MyDatabase/</Address>
        <!-- endif -->

        <TransportType Name="SQL" Capabilities="81163" ConfigurationClsid="6e688659-1374-4f37-90dc-40cb6dd3c73d" />
        <!-- ifdef ${_xml_preprocess} -->
        <!--  

        <TransportTypeData>
         <CustomProps>
            <AdapterConfig vt="8">
              <Send>
                <connectionString>${MyDatabaseConnectionString}</connectionString>
                <outputRootElementName>MyRootElement</outputRootElementName>
                <documentTargetNamespace>http://MyNameSpace</documentTargetNamespace>
                <uri>SQL://($MyDatabaseSQLServer)/${MyDatabaseDatabase}/</uri>
              </Send>
            </AdapterConfig>
          </CustomProps>
        </TransportTypeData>

        <!-- else -->
            <TransportTypeData>&lt;CustomProps&gt;&lt;AdapterConfig vt="8"&gt;&amp;lt;Send xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"&amp;gt;&amp;lt;connectionString&amp;gt;Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=MyDatabase;Data Source=(local)&amp;lt;/connectionString&amp;gt;&amp;lt;outputRootElementName&amp;gt;SQLInsertCommonErrorResp&amp;lt;/MyRootElement&amp;gt;&amp;lt;documentTargetNamespace&amp;gt;http://MyNameSpace&amp;lt;/documentTargetNamespace&amp;gt;&amp;lt;uri&amp;gt;SQL://(local)/MyDatabase/&amp;lt;/uri&amp;gt;&amp;lt;/Send&amp;gt;&lt;/AdapterConfig&gt;&lt;/CustomProps&gt;</TransportTypeData>
        <!-- endif -->        

Issues with this pattern:

  1) After running Element Tunnel, you have to carefully copy in the non-tunneled original <TransprotTypeData>.   However, if the else is never being used, I was just wasting a lot of time doing that.

 

3. Elimination of the "else" within the "ifdef": 

        <!-- ifdef ${_xml_preprocess} -->
        <!--     <Address>SQL://${MyDatabaseSQLServer}/${MyDatabaseDatabase}/</Address> -->
        <!-- endif -->

        <TransportType Name="SQL" Capabilities="81163" ConfigurationClsid="6e688659-1374-4f37-90dc-40cb6dd3c73d" />


        <!-- ifdef ${_xml_preprocess} -->
        <!--  

        <TransportTypeData>
         <CustomProps>
            <AdapterConfig vt="8">
              <Send>
                <connectionString>${MyDatabaseConnectionString}</connectionString>
                <outputRootElementName>MyRootElement</outputRootElementName>
                <documentTargetNamespace>http://MyNameSpace</documentTargetNamespace>
                <uri>SQL://($MyDatabaseSQLServer)/${MyDatabaseDatabase}/</uri>
              </Send>
            </AdapterConfig>
          </CustomProps>
        </TransportTypeData>

       <!-- endif -->        

Issues with this pattern:

  1) No real big issue, but next pattern seems even easier.

 

4. Super wrapper - one big "ifdef":

        <!-- ifdef ${_xml_preprocess} -->

             <Address>SQL://${MyDatabaseSQLServer}/${MyDatabaseDatabase}/</Address>

        <TransportType Name="SQL" Capabilities="81163" ConfigurationClsid="6e688659-1374-4f37-90dc-40cb6dd3c73d" />

        <TransportTypeData>
         <CustomProps>
            <AdapterConfig vt="8">
              <Send>
                <connectionString>${MyDatabaseConnectionString}</connectionString>
                <outputRootElementName>MyRootElement</outputRootElementName>
                <documentTargetNamespace>http://MyNameSpace</documentTargetNamespace>
                <uri>SQL://($MyDatabaseSQLServer)/${MyDatabaseDatabase}/</uri>
              </Send>
            </AdapterConfig>
          </CustomProps>
        </TransportTypeData>

       <!-- endif -->        

This is the way I'm leaning to try today.  It's more clear if I have 5 SQL ports  I would just put one IfDef at the top of the binding file (and the closing "endif" at the bottom),
and let the substitution happen everywhere.

5. The next step would be to even eliminate the ifdef at the top and bottom if possible. 

6. The ultimate step would be what I mentioned above, where we could just export our binding file, run the reverse XML Preprocess, and voila.

 

Anyway, to summarize, I love not having to include the "ELSE" and it never registered with me that the requirement to use it was lifted.

Neal

 

 

 

Coordinator
Jul 23, 2010 at 10:17 PM

I have some good news for you!  In the next release you'll be able to set a new property, RequireXmlPreprocessDirectives, to false in order to achieve #5 above -- no #ifdef/#endif's at all.

As for ease of use, binding files are definitely the hardest part of configuration.  However, the defaults are set up in the Deployment Framework such that configuring a new BizTalk project of average complexity should not take more than 10-15 minutes.

First, you use the Add New Project wizard.  It gives you your entire project structure.  Next, you edit the generated .BTDFPROJ to specify the ItemGroups that define your project artifacts and their locations.  The last step is creating a PortBindings.xml (master bindings and ElementTunnel are disabled by default), which you can just export from BizTalk Admin once it is manually configured for the first time.

After it's up and running properly, then you can branch out and start creating a PortBindingsMaster.xml, adding settings to the spreadsheet, etc. if desired.

For most situations, I think the new RequireXmlPreprocessDirectives will be a great option.  It definitely simplifies adding setting macros to the binding file (and any other XML files that are being XMLPreprocess'ed).

Thanks,
Tom