Apr 282009
 
This is the second installment in my series called “Field Notes.”

This is where I share what I learn in the trenches with you.  I believe it also helps demonstrate the breadth and details of the work I do.

As with most client projects, it all starts from an email.  Due to a Non-Disclosure Agreement (NDA), some information has been removed, but the client is very interested in me getting this information out, so many thanks to them for providing me this opportunity.  What has been removed doesn’t change the benefit of the notes or the value of the traps that have been unsprung!

Also, I hope you can see that I work from “requirements via email” as well as formal requirements documents and that I can also sprint 🙂

EMAIL:

Overview:

The Journey Recapped in an Executive Overview

    The below bullets 1-4 cover the key points of the three phase project “Salesforce Contact attachment sent to. ”  This was a proof of concept developed by Grigsby Consulting LLC/David W. Grigsby sponsored by client.  This project ran from Monday 4/13/2009 to Monday 4/20/2009.  It was a key demo and fact finding mission for a client meeting 4/21/2009.
    In the first phase of the project, David’s approach was via traditional Object Oriented Development practices and endeavored to leverage the Salesforce preferred WSDL Apex creation that the Salesforce promotes for integration of external application endpoints and services.  This path failed and that phase completed 4/18/2009 late in the afternoon after confirmation from Salesforce Premier Support that this approach was not currently supported.
    David, without being asked then started phase two and went at the problem from a very low level.  He attacked it from both at the wire and the code.  He captured a live XML SOAP package from the API Integrated Sample and coded that capture XMP SOAP Message with header and payload into a Salesforce class method.  He then coded a http request created from strings that created a Package that showed in the System console with success by late that Saturday Night.  So now the Salesforce/ integration had two approaches, one not supported and one supported but very low level.
    David was asked by client if he could complete the demo by Monday and was given a lets go for it by David Sunday, which started phase three.  By 2:30pm PST David and client had it working.
    Grigsby Consulting LLC agreed to do this project for $X knowing it would be a great challenge and provide insight and experience into many of the technologies and platforms of interest to all involved parties.  As a reference point only, had this been a time and materials proof of concept project and fact finding mission, it would have cost $9440.  This is based on the 93 hours used, billed at Grigsby Consulting LLC best client retainer rates.  Grigsby Consulting LLC is very happy, excited and pleased with the results and thanks client management for the opportunity to serve and provide David W. Grigsby on such an interesting and exciting project.
    Grigsby Consulting LLC looks forward to future opportunities where we can mutually benefit when results matter so much and yet so little is known.  I know we all crave complete definition of the problem space but I think this example shows that sometime an email requirement doc with a “bounty” is appropriate to make progress.

1. The Project Request
a.

     Create an integrated demo in SF using Apex code to send a SF attachment to Client API per client’s email requirements

2.  The Goal
a.

      From SF pass a PDF stored in SF via SF Apex to Client API Endpoints via SOAP by sending Package via CreateandSend API without using an external third party between

3.  The Hurdles
a.

      WSDL Apex Class creation Limitations
          -Size of final Apex Class – 100K characters
          -Size of WSDL input – 1Meg
          -No Imports and no implied types from defined named spaces if they contain definitions (xsd which ws-security does).
          -No attributes (xsd which ws-security does).
          -No annotations (xsd which ws-security does).
          -No multiple bindings (which WSDL does as most do 1 and 12)

b.

      Concrete examples of how to fill the WSDL generated Classes for ws-security SOAP header with List classes to be rehydrated via the WebServiceCallout.Invoke SF platform method (have a SAP output example, but still vague)
          – Salesforce Premier Supports states it is not supported per Case XXXXXXXX – CRITICAL: Getting WebServiceCallout.Invoke() to Deliver

c.

      Step level debugging with breakpoints and variable inspection
          – Neither Eclipse with the Force.com Perspective nor the System Log via SF allow for this
          – Very pleased with System.Debug output and Debug logging by user, so was better than first blush and did help immensely

d.

      Documentation on every level limited or incomplete concerning WSDL SOAP Headers and ws-security

4.  What I know that needs to be changed in the current demo to head to a product Alpha/Tech Preview/Beta
a.

      Create XML Classes for SOAP to eliminate Strings (as always String concatenation is way too heavy and significantly slows module – But works!)


b.

      Decide if WSDL overhead is worth bloat
          -I don’t think so, clean XML doc with element replacement of values should probably be tightest and cleanest, but I would try to validate my intuition by have both working – may want WSDL for other platforms

c. 

    Factor Classes to one Client API call per Class

d.

      Create test classes/harnesses for all Client API Classes

e.

      WSDL – Complete self contained Version
          – All APIs in one
          -One API only without other unneeded elements, types, messages, ports and bindings

f.

        Asserts

g.

      Try Catches

h.

      Handling of responses from webservice call response
         -Need to document OK-200, BAD-400

i.

          Comments

j.

          [removed]

l.

          [removed]

m.

       Sending Status message window (partly coded with Loading gif)

n.

        Data validation after data selection
          – contact with valid email

o.

      [removed]

p.

      Document Client API usage of ws-security, what is used, what is not and whether that belongs in the WSDL output from the API or just in the SDK/Forum
          – Does the Nonce need to change?
          – What is the valid lifetime of a ws-security header (in demo it is till July 2009)
          – Does Client API use the message and reply?

q.

      Quality Requirements, design, developer and test plan docs besides product help docs 🙂

Why This Happens:

    This is development, whether it is on embedded devices, on premise applications or cloud based applications.  We want to reduce and integrate these desperate software applications on these different systems into one seamless business process for our end users.  Everyone believes their documentation, SDK’s and kb articles have the 80% of the product use cases covered, and I would agree with that.  It is then left to the community forums and blogs to cover that last 20%.  Normally that 20% of use cases are focused on edge/corner cases or Next version features that couldn’t make it into the product due to testing, time, marketability or perceived client need.  It is at times like this when consultants get to do a real service and find the boundaries for clients and document them for others to understand and for product teams to see how clients are pushing the boundaries of what their product does and what is supported.  It really is the best way and is very much like how commercial and open source software evolves and matures.
    The

ws-security

    SOAP headers (web services [-] security) are widely used, the Internal Revenue Service (IRS0 – both federal and state.  The OASIS Web Services Security (WSS) technical committee maintains this standard and it edited by Microsoft, IBM, VeriSign and Sun with many contributors.  The

PDF documentation

    is a solid one.  One major benefit is that it is sessionless, aka you don’t have to “Login” or “Logout” before or after a web service call, as you provide your credentials as tokens and keys over HTTPS each and every time, so overhead is very light and sessionless/stateless.
    The Web Service Description Language (

WSDL

    ) is a language for describing web services in Extensible Markup Language (

XML

    ) that is widely used and is evolving and uses Hypertext Transfer Protocol (

HTTP

    ) and Simple Object Access Protocol (

SOAP

    ) protocols to get the work done, again very widely used 🙂
    Object oriented programming languages that leverage libraries and helper classes for the platforms being used.  This is a preferred development practice, but sometime you must get down and dirty to understand it and get past areas not documented or supported in the platform you are integrating with.
    Developers use Languages, Software Development Kits (

SDK

    ) and Standards to create applications or application integrations like the one I was asked to create between Salesforce and in the cloud.  <
    So I was armed with the following:
  • Salesforce APEX Language
  • Salesforce Developer and Test environments with related documentation
  • Eclipse IDE with Salesforce Perspective
  • Client 3.0 SDK
  • Test Environment and related application documentation
  • WS-security standard

So my job was normal in that I was to take all that and my clients simple email requirements using all the aforementioned protocols and languages to make some end users life easier by reducing the number of “cuts and pastes” needed to send a document from Salesforce to by putting all that into one button inside Salesforce.  My, what a tasty project recipe!

What I found out was that there was very little real concrete WSDL out there with ws-security because rarely do you have to create it, as languages like java and .net have system libraries (WSE) to handle getting it added to your web service class.

Second issue was that WSDL, like many XML languages use imports and contains complex hierarchies to express complex data types, elements and messages.  Salesforce WSDL2APEX requires it be a simple self contained WSDL.

On top of that, none of the tools, XMLSpy, Eclipse or Visual Studio will auto collapse multiple schemas into one.  That was a manual process that required 14 passes to get it right.  Make change to the WSDL via Atlova XMLSpy to fix the error reported from WSDL2APEX, try it again, rinse and repeat.

Now of course I, like you, am finding out all of this while trying to get the project setup, working and meet a hard deadline.  The setup of the working environment takes 6-8 hours the first time.  Fortunately I am my own IT staff, so low overhead and lots of experience.  Then the reading up and learning the ws-security standard, the APEX Language output from the WSDL, and matching all this up with the 3.0 API SDK for the web service calls I needed from the API Integrated Sample example in .NET on how to use the 3.0 API.  I did find one WSDL post that was pretty much there, except for the Import, but I just added in the needed XSD from both ws-security and ws-utility so I could remove the imports.  Thanks goes out Hari Kolasani and his post back in 2004.  I found lots of “Help I can’t make it work posts” about WSDL and WSDL2APEX on the force.com discussion forums and the web.  I am a great Googler, but nowhere did I find a “here you go, this is what it looks like and this is how it works,” so I am creating this for the next person.  Just call me Daniel Boone.
Approach and Traps:

GOAL:

    Need to make a web service call from inside a Salesforce APEX class to an API web service endpoint and implement features from client email

1.

     Setup Eclipse IDE with Salesforce Perspective (

Well Documented

    )

2.

     Download Salesforce Source Apex Code from Dev/Test Environment from Eclipse

    a.

       review attachments in contacts and see how to add button to call my new class that will call the web services – many

    examples

      and here is a great

    blog

      on Salesforce code ups.

    3.

       Setup Visual Studio IDE with the Client API integrated sample

      a.

         Validate it is working with test account

      4

        . Generate Apex based WSDL class from client 3.0 API SDL WSDL called client_3.wsdl using WSDL2APEX

        a.

            Major Tom we have a problem

          b

            . To get your WSDL to even get generated a class the WSDL must not:
                 -Generate a final Apex Class greater than 100K characters
                 -Size of WSDL input must be less than 1Meg
                 -No Imports and no implied types from defined named spaces if they contain definitions (xsd which ws-security does).
                 -No attributes (xsd which ws-security does).
                 -No annotations (xsd which ws-security does).
                 -No multiple bindings (which WSDL does as most do 1 and 12)

          Issue

            Use the Salesforce Client API L2APEX class generator to create classes to call that will evoke web Services using ws-security SOAP headers
            SF WSDL Apex ws-security brief

          PROBLEM AS UNDERSTOOD: XML rehydration via WebServiceCallout.Invoke() of WSDL2Apex generated Apex Classes from WSDL for SOAP headers is undocumented.

          In the first phase of the SF2Client API project, David’s approach was via traditional Object Oriented Development practices and endeavored to leverage the Salesforce preferred WSDL2Apex creation of Apex Classes from WSDL that Salesforce promotes for integration of external application endpoints and services.  This path failed after confirmation from the Salesforce Premier Support “Case XXXXXXXX – CRITICAL: Getting WebServiceCallout.Invoke() to Deliver” conclusion that this approach was not currently supported.

          It is my belief this is a misunderstanding of supported boundaries on part of SF support from what we wanted to do compared to how to go about it.  It is the difference of requesting internal documentation on XML rehydration via WebServiceCallout.Invoke() vs. formal SF support of ws-security in the SOAP header by standard SF classes.  I say this as I was able to get WSDL2Apex to generate the classes without errors and compile that had the SOAP header element “Security”.

          I believe after another 40+ hours of my time I would have been able to reverse engineer how it was rehydrating the XML by capturing the output to the endpoint at my server, but still feel this was not what was asked of me.  Now if WebServiceCallout.Invoke() then I also believe it would be a waste of time as I am sure the best documentation is the source code and the developers that created and maintain WebServiceCallout.Invoke() .  This is why I went with a HTTP request whereby I formed the XML via strings instead of classes to send to the Client API endpoint to make it work correctly.  Before using the WSDL2Apex generated classes I was getting the error ” Current Error: Web service callout failed: WebService returned a SOAP Fault: Security requirements are not satisfied because the security header is not present in the incoming message.  faultcode=q0:Security faultactor=https://demo.clientapi.net/api/3.0/api.asmx”

          I believe we could collaborate with SF Dev support in order to discuss, understand and document the parameter (XML elements, attributes and values) rehydration (marry up) of the WebServiceCallout.Invoice() Apex platform call parameter string filled from the WSDL Generated Classes for the SOAP header and body elements.  The ws-security element Security contains the ws-security elements user, account, password, but based on the classes generated, I was not able to readily discern how they were to be populated with the proper element/attribute names and values.

          The only document that I found that comes close is http://wiki.developerforce.com/index.php/Apex_Web_Services_and_Callouts and below is an excerpt:

          “Debugging WSDL2Apex:As mentioned earlier, the key benefit of using WSDL2Apex is to let Apex handle all of the hard work of invoking a web service.  But during development, it is often key to debug your Apex logic including getting the output of the Apex generated SOAP XML messages (requests and responses).  ”

          In the end, it is my believe the SF community and developer ecosystem needs concrete examples of how to fill the WSDL generated Classes for ws-security or any other SOAP header element with List classes to be rehydrated via the WebServiceCallout.Invoke SF platform method .  I did find a SAP output example, but still very vague as they didn’t provide the injection code for the classes, just the output.  It was going to be a starting point for my reverse engineering if I was asked.  I am happy to be that person but I would like to reduce the burden with SF help.

          Trap

            Not Supported currently by Salesforce but may work once enough information is documented and/or Salesforce

          Trap unsprung safely

            Okay, let’s just generate SOAP XML from strings old school and send via a basic HTTP Request by passing WSDL and the generated classes.
            So I attacked it from both at the wire and the code.  I captured a live XML SOAP package from the API Integrated Sample by modifying it to send to a web server endpoint I coded up to capture it raw and save it to a file.  I then coded that captured XMP SOAP Message with header and payload into a Salesforce class method.  I then coded a http request created from strings that created a Package that showed in the System console with success.
            So now had two approaches, one not supported and one supported but very low level.

          5.  Now get the document as a PDF stream and convert to base64 and replace in the string where the PDF stream goes.

          6.  Complete the rest of the feature wire to get a working demo with less than 26 hours left, no pressure 🙂

          7.  Success!

          Tools Needed:

          a.

            Salesforce.com

                   -SF
                   -SF Developer Org
                   -System Log
                   -Eclipse IDE Perspective
                   -Language – Apex

            b.

              Eclipse – Desktop Development Environment

                     -Version 3.3.2
                     -Force.com Perspective
                     -SF Security Token

              c.

                Client SDK

                       -3.0 SDK –
                       -API Integrated Sample .NET Application – Validation and debugging of SOAP XML emitted and see how it was sending ws-security – very much the same as java – platform supported library calls
                       -Java app – review how it was sending ws-security – very much the same as .net – platform supported library calls
                       -Demo System Console – confirmation Package, validation of working Package, PDF, Tag locations, contents and behaviors, custom Fields

                d. Microsoft Visual Studio 2008

                       -Run API Integrated Sample and modify to send final call to local web server page to capture actual SOAP XML being sent to Client API Ping/CreateandSend API calls
                       -Validate XML from output of system.debug from SF system.log output
                       -step level debugging of Sending and Receiving of API Integrated Sample and local Host Capture

                e

                  .

                Atlova XMLSpy

                       -Create all in one WSDL xml – couldn’t be done with Visual Studio – didn’t visually or otherwise distinguish what types where in the name space and what was imported

                How to disarm the trap:

                1.

                    Create Page on Local web server to capture incoming HTTP Request to File

                a.

                    ReceivePost.aspx and ReceivePost.aspx.cs

                2.

                    Modify API Integrated Sample SDK Sample to send to Local web server (edit web.config)

                3.

                    Create Apex class with method to assemble a string from the captured file SOAP XML in smaller string fragments and send to the API to validate it is working and get a System.HttpResponse[Status=OK, StatusCode=200] response via the System Log or debug log.

                4.

                    Run the Class method via System Log immediate execute and watch it work!

                Additional Considerations and lessons learned:

                a.

                    Learn faster by failing even faster 🙂

                b.

                    Start basic low level first, instead of last
                       -use supported helper classes and technologies after proof at wire level is working

                c. 

                  Not believe anyone else has solved using ws-security with a self contained WSDL without using a language with system/platform helper api’s/libraries’ and had time to post a solution – Found one close, but not with a self contained WSDL

                d.

                    Restate Question to SF Premier Support into 2 parts
                       – same ws-security question but with follow-on
                       – how is (provide sample code of) the List used for the loading of the WSDL created support classes done correctly so that they can be rehydrated via the WebServiceCallout.Invoke SF platform method (have a SAP output example, but still vague)

                Result:

                Wonderful client email, this blog post, many more on and Force.com and of course billable time 🙂

                Resources:

                1.  Web page ReceivePost code to capture XML

                ReceivePost.aspx

                ReceivePost.aspx.cs

                2.  Captured CreateandSendPackage Soap Call

                3.Final Apex Class TestPing with DoPing() method

                Using Ping instead of CreateandSendPackage for brevity – Just replace SOAPXMLBody string with the above and change your Username, Password and AccountID with yours.  You will need to break out over several string concatenations aka

                SOAPXMLBody = SOAPXMLBody + “big long string”;

                4.  Immediate execution code from Salesforce System log or Eclipse

                SalesForce System Log Debug Output


                Debug output should look like:

                5.  Final WSDL for Ping and CreateandSendPackage