November 20, 2010

Tolerado WSC now gives transparent batcher support for Salesforce Partner APIs !

One can’t persist more than 200 records in one web service call using Salesforce Partner calls like create()/update()/upsert(). You will stumble upon this error on such attempts

“EXCEEDED_ID_LIMIT: record limit reached. cannot submit more than 200 records into this call”

The solution to this is to use a batching mechanism as described in the Salesforce Cookbook here : http://wiki.developerforce.com/index.php/Batch_records.java. This batching recipe code sample is giving simple abstraction to batching for update/create calls and is really easy to port to your production app, because its not dramatically changing the code design.

Transparent Batching available in Tolerado for WSC !

I thought Tolerado for Salesforce WSC should also make best use of this delicious Batching recipe. So, the same Batch_records.java batching is ported to Tolerado PartnerStubs. Here are highlights of this port :

  • Batching available to all create()/update() and even upsert() calls.
  • Batching is made available transparently. Being transparent here means, client code using ToleradoPartnerStub will not change at all, “batching will come into play automatically if number of records are more then 200

Updated Tolerado Jars are available in download area

For those who want more fine technical details on batcher implementation, please check these classes:

Your views ?

Would love to hear your views/suggestions/feedback on this !

November 14, 2010

Tolerado for Salesforce WSC release 1.1 out !

A major release 1.1. is done for Tolerado for WSC  APIs today.

Here is a quick list of changes that contributed to this release, like

  • Added support to use all Tolerado stubs with only sessionId and serverUrl in hand. Previously username/password was a must. (Issue 1)
  • Fixed cache sharing issue between enterprise and partner sessions (Issue 2)
  • Added support to use either Product, Developer, Sandbox, Prerelease or Custom Org Type(Enviornment) with Tolerado. Now one can feed in either the Enviornment or the hostname to use for authorizing with Salesforce.
  • Added support to specify Salesforce API version to Tolerado stubs. That means more control if you still want to stuck with previous API version.
  • Tolerado Sobject, internal changes for more flexiblity in extending and changing behavior like caching.
    • Added new method List<XmlObject> getChildrens(String relationshipName)
    • Added final getType(), setType() methods, to get/set SObjectType.
    • Deprecated setAttribute(String attribName) to match with getters like getValue(String) getTextValue(String) etc, we are promoting setValue(String, Object), This method(setAttribute) might be removed in future releases.
  • Simplified design of Tolerado stubs, to facilitate easy extensions.
  • Removed getLoginDriver() from ToleradoSession API contract.
  • ToleradoMetaStub for metadata wsdl, has new recoverable calls like
    • AsyncResult checkStatus(final String asyncProcessId)
    • RetrieveResult checkRetrieveStatus(final String asyncProcessId)
    • AsyncResult deploy(final byte zipFile, final DeployOptions deployOptions)
    • DeployResult checkDeployStatus(final String asyncProcessId)
  • Added logout() call to both Tolerado Partner & Enterprise stubs.

Complete release notes available here.

Any feedback or issues with this new release ? please let me know !

Salesforce WSC added support for Session Timeout handling !

Retrying web service call failures/exceptions is important, because not every exception means its “end of world”, their is always some HOPE, for some exceptions like Session Timeout / Invalid Sessions Id. One can always renew the session (if credentials are available), and make those failing web service calls work again.

Want to know more about such recoveries, its explained in this previous post “Why WS-Client should RETRY Web Service Exceptions ?”.

SFDC WSC API & Session Renewal !

Salesforce WSC(Force.com Web Service Connector) recently in release 20 introduced the same concept of renewing session on timeout or invalidation. Unfortunately, I can’t find this anywhere in any official WSC documentation, I came to know about this new feature, when I was debugging some error and stumbled upon WSC ConnectorConfig class. This class has a new attribute called “sessionRenewer”, SessionRenewer is a Java interface, implementations of which can be used to renew session in case of Session-Timeout. These SessionRenewer implementations can be optionally set by calling the relevant setters in ConnectorConfig i.e. setSessionRenewer(SessionRenewer sessionRenewer);

My “FAILED” experiment with WSC SessionRenewer Sad smile !

I thought of giving this new feature a try and created following test class. I was not able to make this code work well with SessionRenewer. I think SessionRenewer just handles session timeouts, not a generic INVALID_SESSION_ID fault raised because of many other reasons. So, to know where I was going wrong, I raised the issue on SFDC WSC Google code project. If any one of you was able to successfully use SessionRenewer, please let me know.

public class WSCSessionTest {
 public static void main(String[] args) throws Exception {
  ConnectorConfig cfg = new ConnectorConfig();
  cfg.setUsername("sfdcusername");
  cfg.setPassword("sfdcpass");
  // set the SessionRenewer implementation
  cfg.setSessionRenewer(new WSCSessionRenewer());
  PartnerConnection binding = Connector.newConnection(cfg);
  // force logout, to see if session timeout,
  // because of "invalid session" comes
  binding.logout();
  // Try querying now
  QueryResult qr = binding.query("Select Id, Name from Contact limit 1");
  System.out.println(qr.getRecords()[0].getField("Name"));
 }

 public static class WSCSessionRenewer implements SessionRenewer {
  @Override
  public void renewSession(ConnectorConfig config)
    throws ConnectionException {
   System.err.println("Renewing Session for Config : " + config);
   config.setManualLogin(true);
   PartnerConnection binding = Connector.newConnection(config);
   LoginResult lr = binding.login(config.getUsername(), config.getPassword());
   
   // I am updating the config, back as I don't have reference to the original
   // PartnerConnection object.
   config.setSessionId(lr.getSessionId());
   // If using Partner or Enterprise WSDL 
   config.setServiceEndpoint(lr.getServerUrl());   
  }
 }
}

Tolerado (WSC or Apache Axis) users already have all this !

For those who don’t know what is Tolerado,

“Project "tolerado" is a Java based WS client framework for better and fault tolerant use of Web Service APIs given by Salesforce. Tolerado-WSC works on top of highly performance SFDC Web Service Connector API. In case your project is depending on Apache axis 1.4, you can use Tolerado for Axis

Tolerado framework, already gives all this session renewal and many other forms of error recovery transparently. So developers who ported their Java WS projects to Tolerado, don’t need to worry about retryable/recoverable web service error handling. They can just focus on grooming the integration and business logic with Tolerado.

To even more simplify developers life, I requested WSC team, to integrate Tolerado into the WSC library. I raised this issues http://code.google.com/p/sfdc-wsc/issues/detail?id=25 long back on WSC Issue list. I am still waiting to hear from WSC team, please let me know.

Conclusion

Its good to see this effort from WSC team, to add recovering abilities to the API. But renewing session on session timeout or invalidation is one thing, there are many other failures (explained in this post), that can be retried and recovered in same way. So, I expect more of such retryable/recoverable error handling coming to WSC in near future.

Also, SFDC WSC team : Please add API “release notes” page some where on WWW !

November 10, 2010

HTTP Compression with Apache Axis, a huge performance boost !

As mentioned in my previous post, I was researching/benchmarking on how HTTP Compression with Apache Axis improves performance. The results I got post HTTP compression, are surprisingly amazing i.e. “response time of web service calls is reduced to one third and one forth in many cases”.

Though HTTP Compression is a generic topic, not related to Salesforce, but for my benchmarking, I used Salesforce SOAP web services.

Good news : I got a highly noticeable response time difference with each Partner, Enterprise, Apex and Metadata web service calls.

How to make HTTP Compression work in Axis (Salesforce)

Prerequisite – setup Classpath with dependencies

Apart from WSDL2Java classes for your partner, metadata etc WSDL. You need following jars in classpath.

  • axis.jar (I used Axis v1.4)
  • commons-httpclient-3.1.jar 
  • common-codec-x.jar (If you are getting this error http://bit.ly/9dBisn)
  • Other jars that are usually included when working with Salesforce Webservice + Apache Axis 1.4.  Like jaxrpc.jar, commons-lang-x.x.jar, commons-discovery-x.x.jar, wsdl4j-1.x.x.jar. All these dependencies are available with your axis 1.4 binary download. Check the “axis-1_4.zip\lib” folder.

Step 1 - Extend Axis Service to @override createCall() method

As explained in this post by Simon. We need to extend the WSDL2Java generated extension class to “org.apache.axis.client.Service” as shown below. In case of Salesforce partner.wsdl, this class is SforceServiceLocator.

import javax.xml.rpc.Call;
import javax.xml.rpc.ServiceException;
import org.apache.axis.transport.http.HTTPConstants;
import com.sforce.soap.enterprise.SforceServiceLocator;

public class SforceService extends SforceServiceLocator {
   public Call createCall() throws ServiceException {
      Call call = super.createCall();
      call.setProperty(HTTPConstants.MC_ACCEPT_GZIP, Boolean.TRUE);
      call.setProperty(HTTPConstants.MC_GZIP_REQUEST, Boolean.TRUE);
      return call;
   }
}

If you are working with other Salesforce WSDLs, then you have extend these classes in the exact same way. 

  • Apex WSDL : com.sforce.soap._2006._08.apex.ApexServiceLocator
  • Enterprise WSDL : com.sforce.soap.enterprise.SforceServiceLocator
  • Metadata WSDL : com.sforce.soap._2006._04.metadata.MetadataServiceLocator

Note: We just need to change “extends SforceServiceLocator” in above code snippet to these classes, rest of the code will remain the same.

Once you are done with the change then you client code needs, to use this extension class for creating the stubs, as shown below

SoapBindingStub stub = (SoapBindingStub) new SforceService().getSoap();
// prepare the stub by either making a login call 
// OR
// by setting headers for SessionId/ServerURL etc
QueryResult qr = stub.query("Select Id, FirstName, LastName from Contact");

// Similarly you can create stubs for Apex, Metadata and Enterprise WSDL 
// as shown below
// Apex WSDL
ApexBindingStub apexBinding = (ApexBindingStub) new SforceService().getApex();
// Metadata WSDL
MetadataBindingStub apexBinding = (MetadataBindingStub) new SforceService().getMetadata();
// Enterprise WSDL (looks same as Partner, but SoapbindingStub here is 
// under package com.sforce.soap.enterprise.SoapBindingStub
SoapBindingStub stub = (SoapBindingStub) new SforceService().getSoap();

Step 2 - Tell Axis to use “CommonsHTTPSender” instead of “HTTPSender”

One way to do that is create a config file named “client-config.wsdd”, as explained by Simon. Here is the relevant text from his blog post :

“you have to configure axis to use the vastly superior CommonsHTTPSender class rather than the default HTTPSender class as its transport, this is done by changing the transport entry in the client-config.wsdd file from java:org.apache.axis.transport.http.HTTPSender to java:org.apache.axis.transport.http.CommonsHTTPSender and adding Commons HTTP to the class path.”

You can Google around how to create a client-config.wsdd file for this. I usually hate creating config files, so here is an alternate approach if you want to achieve the same using following Java code (Add this code inline to some class that’s called on application startup or prior making SOAP calls)

// You need to include this static block
// in some startup class or just set the AxisProperties
// some where else.
static {
   AxisProperties.setProperty(
     EngineConfigurationFactory.SYSTEM_PROPERTY_NAME,
     CompressedEngineConfigurationFactory.class.getName());
}

public static class CompressedEngineConfigurationFactory implements
  EngineConfigurationFactory {

  public static EngineConfigurationFactory newFactory(Object param) {
    return new CompressedEngineConfigurationFactory();
  }

  public EngineConfiguration getClientEngineConfig() {
    BasicClientConfig cfg = new BasicClientConfig();
    cfg.deployTransport("http", new SimpleTargetedChain(
      new CommonsHTTPSender()));
    return cfg;
  }

  public EngineConfiguration getServerEngineConfig() {
    return null;
  }
}

That’s it guys, try making calls now. You have HTTP Compression “ON”.

Step 3 - Feeling the response time difference

To really feel, “how SOAP compression is great”, try benchmarking Salesforce web service calls that return huge response. For ex. a SOQL query that returns 2000+ records, or a heavy metadata call.

When not to use HTTP Compression ?

Http compression(GZIP) is good for web service calls working on “TEXT” request/response. If your web service call is returning something binary like ZIP files. I would suggest, don’t use HTTP compression on that, because that is already compressed.

So ideally, one needs to decide “When to turn on HTTP compression ?”, based on Web service Response Type.

Do “SFDC WSC API” needs HTTP Compression ?

What is SFDC WSC API ?

SFDC WSC API i.e. The Force.com Web Service Connector (WSC) is a high performing web service client stack implemented using a streaming parser. WSC also makes it much easier to use the Force.com API (Web Services/SOAP or Asynchronous/BULK API).

Http Compression with WSC !

Good news here is that, HTTP Compression is “ON” by default with WSC. So one needs to do no tweaks to get it with WSC. I must say, great job done by WSC team, the API is very easy to use and gives many hidden cool stuff !

If required, you can easily turn HTTP Compression “OFF” by calling

com.sforce.ws.ConnectorConfig.setCompression(false);

 

What do you say ?

Let me know, how Axis compression works for you. Would be interested in getting your feedback on this..!

November 8, 2010

HTTP Compression with Apache Axis for Salesforce Web Services !

Though, I am speaking a lot about using WSC above Apache Axis, but today I found an interesting old post about Http compression with Apache Axis. This is interesting because, it can boost performance for those who are integrating with Salesforce web services using Java Apache Axis. Here are some benchmarks copied Salesforce WIKI :

“Your request to the server is gzipped compressed (the MC_GZIP_REQUEST property), and the request advertises (via the Accept-Encoding http header) that it'll accept a gzipped response (the MC_ACCEPPT_GZIP property). In the case of an sforce call like Query which can return very large responses you can see some big decreases in message sizes, a quick test here shows that an uncompressed response of 376426 bytes compresses down to a measly 33617 bytes (approx 9% the original size)”

These results look great !

I haven’t tested it, but will start soon. Here are the pointers, if you want to start and give some feedback.

Though it seems there are couple of problems, as mentioned in the WIKI article and force.com blog comments. Still, I am sure this should be doable and stable now, as all of those comments are of year 2005 and SFDC API v6, we are on SFDC API v20 now Smile

Anyone already had hands on experience with this compression ? if YES, please share. I will write a followup post soon, about how my experiment with compression goes.

November 2, 2010

Understanding–“No such column on entity 'Name'” Error !

One can get errors like

“No such column 'SmallPhotoUrl' on entity 'Name'.” OR “No such column CompanyName' on entity 'Name'.”

on a SOQL statements like

  • SELECT CreatedBy.CompanyName, FeedPost.Body, Id, CreatedBy.Name FROM AccountFeed
  • SELECT CreatedBy.SmallPhotoUrl, FeedPost.Body, Id, CreatedBy.Name FROM AccountFeed
  • SELECT Id, FeedPost.CreatedBy.SmallPhotoUrl, (SELECT Id, CreatedBy.Name, CreatedBy.SmallPhotoUrl, CommentBody FROM FeedComments) FROM NewsFeed

The column responsible for this error is bold highlighted in above SOQL statements. The error is so confusing, you must be wondering that : “I queried on User(CreatedBy) relation, then why the field is expected to be their on entity ‘Name’ ?

Why this error is thrown for entity “Name” ?

Its because “CreatedBy” and many other Salesforce relations become “Polymorphic” if pointing to more than one type of object as parent. This is what Salesforce "Relationship Queries" doc says:

“A polymorphic key is an Id that can refer to more than one type of object as a parent. For example, either a contact or a lead may be the parent of a task. In other words, the WhoId field of a task may contain the Id of either a contact or a lead. If an object can have more than one type of object as a parent, the polymorphic key points to a Name object instead of a single object type.”

So, one can’t expect a single Sobject type fields for polymorphic relations/keys. Rather, check what fields are available in entity “Name”. That is why one can’t find User sobject’s fields like SmallPhotoURL and CompanyName on CreatedBy polymorphic relationships.

Note: Not all CreatedBy relationships are polymorphic it totally varies from sobject to sobject.

How to know if a relationship is Polymorphic ?

To know whether a relation is polymorphic, you can do either of following

  • User describeSObjects call, as described in section “Understanding Polymorphic Keys and Relationships” of Salesforce "Relationship Queries" doc
  • Use Eclipse > Schema browser and expand into the “relationship column > Type Data > Reference To”, as shown below : image