Custom ClaimsAuthenticationManager doesn't get called in WCF REST service

I am trying out a very basic WIF scenario where I have a WCF REST service with its own authentication and authorization managers. The problem is, incoming requests go straight into authorization and completely bypass the authentication manager.

Here is how the two managers and service configurations look

Authentication Manager

using Microsoft.IdentityModel.Claims;

public class AuthenticationHandler : ClaimsAuthenticationManager
{
    public override IClaimsPrincipal Authenticate(string resourceName, IClaimsPrincipal incomingPrincipal)
    {
        return base.Authenticate(resourceName, incomingPrincipal);
    }
}

Authorization Manager

using Microsoft.IdentityModel.Claims;

public class AuthorizationHandler : ClaimsAuthorizationManager
{
    public override bool CheckAccess(AuthorizationContext context)
    {
        return base.CheckAccess(context);
    }
}

Configuration

<configuration><configSections><section name="microsoft.identityModel" type="Microsoft.IdentityModel.Configuration.MicrosoftIdentityModelSection, Microsoft.IdentityModel, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" /></configSections><system.web><compilation debug="true" targetFramework="4.0" /></system.web><system.webServer><modules runAllManagedModulesForAllRequests="true"><add name="UrlRoutingModule" type="System.Web.Routing.UrlRoutingModule, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" /></modules></system.webServer><system.serviceModel><serviceHostingEnvironment aspNetCompatibilityEnabled="true"/><standardEndpoints><webHttpEndpoint><standardEndpoint name="" helpEnabled="true" automaticFormatSelectionEnabled="true"/></webHttpEndpoint></standardEndpoints><behaviors><serviceBehaviors><behavior><federatedServiceHostConfiguration /></behavior></serviceBehaviors></behaviors><extensions><behaviorExtensions><add name="federatedServiceHostConfiguration" type="Microsoft.IdentityModel.Configuration.ConfigureServiceHostBehaviorExtensionElement, Microsoft.IdentityModel, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" /></behaviorExtensions></extensions></system.serviceModel><microsoft.identityModel><service><claimsAuthenticationManager type="Test.WcfRestService.AuthenticationHandler" /><claimsAuthorizationManager type="Test.WcfRestService.AuthorizationHandler" /></service></microsoft.identityModel></configuration>

All requests hit the CheckAccess method in AuthorizationHandler bypassing Authenticate method in AuthenticationHandler. It however works a charm if I change my code/config to use .NET 4.5 security model instead of .NET 4.0 + WIF

Any insight into what could I be doing wrong here will be much appreciated...