Recently in our Microsoft Dynamics CRM 4.0 environment we encountered significant delays in outgoing emailnotifications being sent out from CRM.
An analysis of the Event Viewer logs revealed nothing useful, only the below generic error message
0x80044150 Generic SQL error
I then attached the Sysinternals Procdump process monitoring tool to check for any exceptions – ‘procdump -e 1 -l -f “” Microsoft.Crm.Tools.EmailAgent.exe’. This showed the following error
The remote server returned an error: (500) Internal Server Error
The CRM Email Router service was trying to retrieve emails from the Dynamics CRM Web Service. The above errors were being generated by the CRM Web Service. Anyway, after decompiling the Email Router dll’s and analysing the source code I came across the undocumented setting ‘DebugExceptionInfo‘ in the CRM Web Service. This setting allows you to view the inner exception and stack trace for the exception that occurred within the Dynamics CRM Web Service. I created the Registry String Value ‘DebugExceptionInfo‘ with a value of ‘1‘ in ‘HKEY_LOCAL_MACHINESOFTWAREMicrosoftMSCRM‘ and then re-ran the Procdump utility to monitor for exceptions.
This is what I saw through Procdump BEFORE creating the registry key
[12:26:57] Exception: E0434F4D.System.Net.WebException ("The remote server returned an error: (500) Internal Server Error.")
[12:26:58] Exception: E0434F4D.System.Web.Services.Protocols.SoapException ("Server was unable to process request.")
This is what I saw through Procdump AFTER creating the registry key
[12:44:35] Exception: E0434F4D.System.Net.WebException ("The remote server returned an error: (500) Internal Server Error.")
[12:44:35] Exception: E0434F4D.System.InvalidOperationException ("Client found response content type of 'text/plain; charset=utf-8', but expected 'text/xml'.
The request failed with the error message:
--
System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Data.SqlClient.SqlException: Timeoutexpired. The timeout period elapsed prior to completion of the operation or the server is not responding. At System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()
at Microsoft.Crm.CrmDbConnection.InternalExecuteReader(IDbCommand command)
at Microsoft.Crm.CrmDbConnection.ExecuteReader(IDbCommand command, Boolean impersonate)
at Microsoft.Crm.CrmDbConnection.ExecuteReader(IDbCommand command)
at Microsoft.Crm.BusinessEntities.BusinessProcessObject.ExecuteQuery(CrmDbConnection dbConnection, IDbCommand command, ISqlExecutionContext context)
at Microsoft.Crm.BusinessEntities.BusinessProcessObject.DoRetrieveMultiple(BusinessEntityCollection entities, EntityExpression entityExp, ExecutionContext context)
at Microsoft.Crm.BusinessEntities.BusinessProcessObject.RetrieveMultiple(EntityExpression entityExpression, ExecutionContext context)
at Microsoft.Crm.ObjectModel.CommunicationActivityServiceBase.RetrieveMultiple(EntityExpression entityExpression, ExecutionContext context)
at Microsoft.Crm.ObjectModel.EmailService.BackgroundSendInternal(EntityExpression entityExpression, ExecutionContext context)
at Microsoft.Crm.ObjectModel.EmailService.BackgroundSend(EntityExpression entityExpression, ExecutionContext context, Boolean[]& hasAttachments)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
at System.RuntimeMethodHandle.InvokeMethodFast(Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Web.Services.Protocols.LogicalMethodInfo.Invoke(Object target, Object[] values)
at Microsoft.Crm.Extensibility.InternalOperationPlugin.Execute(IPluginExecutionContext context)
at Microsoft.Crm.Extensibility.PluginStep.Execute(PipelineExecutionContext context)
at Microsoft.Crm.Extensibility.Pipeline.Execute(PipelineExecutionContext context)
at Microsoft.Crm.Extensibility.MessageProcessor.Execute(PipelineExecutionContext context)
at Microsoft.Crm.Extensibility.InternalMessageDispatcher.Execute(PipelineExecutionContext context)
at Microsoft.Crm.Extensibility.ExternalMessageDispatcher.Execute(String messageName, Int32 primaryObjectTypeCode, Int32 secondaryObjectTypeCode, PropertyBagfields, CorrelationToken correlationToken, CallerOriginToken originToken, UserAuth userAuth, Guid callerId)
at Microsoft.Crm.Sdk.RequestBase.Process(Int32 primaryObjectTypeCode, Int32 secondaryObjectTypeCode, CorrelationToken correlationToken, CallerOriginToken originToken, UserAuth userAuth, Guid callerId)
at Microsoft.Crm.Sdk.RequestBase.Process(CorrelationToken correlationToken, CallerOriginToken originToken, UserAuth userAuth, Guid callerId)
at Microsoft.Crm.Sdk.CrmServiceInternal.Execute(RequestBase request, CorrelationToken correlationToken, CallerOriginToken originToken, UserAuth userAuth, Guid callerId)
at Microsoft.Crm.Sdk.Crm2007.CrmService.Execute(Request request)
--- End of inner exception stack trace ---
In the end, the issue turned out to be excessive load on the database causing database operations to timeout. Our DBA’s tuned various database parameters to resolve our issue.
This blog is my place to share my thoughts, helpful solutions and just random nerdy stuff.
Leave a Reply