Spring Batch: Handling exceptions and retrying

In our first post we described how to use Spring Batch for a tax calculation process. However, sometimes we can expect that things go wrong in these processes. In the example of the tax calculation process this could be the actual payment of the taxes. In this blog post we describe two ways how to handle this with Spring’s retry template and exceptions in the Step configuration.

Spring’s retry template

Let’s assume it’s the end of the month and all the companies are paying their taxes. The server gets too many requests and it returns a Http Status Code 503. The same operation may succeed only a couple of seconds later as resources become available again.

In such a case, Spring’s retry library comes in handy. In fact, it just implements the RetryOperations strategy and the simplest general purpose implementation is the RetryTemplate. It could be used like this:

@Configuration
        public class RetryConfig {

            @Value("${employeeJob.taxProcessor.retry.initialInterval:100}")
            private long initialInterval = 100;

            @Value("${employeeJob.taxProcessor.retry.maxAttemptsPerEmployee:3}")
            private int maxAttempts = 3;

            public RetryTemplate createRetryTemplate() {
                Map<Class<? extends Throwable>, Boolean> exceptions = new HashMap<>();
                exceptions.put(TaxWebServiceNonFatalException.class, true);

                RetryTemplate template = new RetryTemplate();
                SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy(maxAttempts, exceptions);
                template.setRetryPolicy(retryPolicy);

                ExponentialBackOffPolicy backOffPolicy = new ExponentialBackOffPolicy();
                backOffPolicy.setInitialInterval(initialInterval);
                template.setBackOffPolicy(backOffPolicy);

                return template;
            }
        }

In this example, inside of our createRetryTemplate , the decision to retry is determined by a SimpleRetryPolicy that allows a retry using a named list of exceptions types (in our case TaxWebServiceNonFatalException). This up to a fixed number of times (maxAttempts).

When retrying after a temporary failure, it often helps to wait a little bit before trying again (in our case having enough resources again on the Tax Payment Webservice backend). If an exception occurs, the RetryTemplate can pause the execution according to the specified BackoffPolicy. The policies provided by Spring Batch all use Object.wait(). A common used case is to backoff with an exponentially increasing waiting period, which can be easily accomplished using the ExponentialBackOffPolicy.

Webservices transactional?

You don’t want to pay taxes to the government twice for the same month. So, the idempotence of this webservice is quite important. The problem with webservices (and of course the one we need to call), is the fact that they may not be transactional or idempotent. This is why we created TaxPaymentWebServiceFacade which wraps the actual web service but makes it (almost) idempotent. If the first invocation fails, the second invocation tries again. But if the first invocation finishes successfully the second invocation does not hit the web service and returns the same result. So basically it retries until it gets a ‘successful’ result.

Now it’s time to use the RetryTemplate we defined earlier to call the web service:

@Component
        @Import(RetryConfig.class)
        public class CallWebserviceProcessor implements ItemProcessor<TaxCalculation, TaxWebserviceCallResult> {

            @Autowired
            private TaxPaymentWebService taxPaymentWebService;

            @Autowired
            private TaxPaymentWebServiceFacade taxPaymentWebServiceFacade;

            @Autowired
            private RetryConfig retryConfig;

            @Override
            public TaxWebserviceCallResult process(TaxCalculation taxCalculation) throws Exception {

                RetryTemplate retryTemplate = retryConfig.createRetryTemplate();
                Callable callable = () -> retryTemplate.execute(doWebserviceCallWithRetryCallback(taxCalculation));

                TaxWebserviceCallResult taxWebserviceCallResult = taxPaymentWebServiceFacade.callTaxService(taxCalculation, callable);

                return taxWebserviceCallResult;
            }

The callable wraps the retry template with the new Java 8 syntax. In this way, the TaxPaymentWebServiceFacade does not need to know anything about Spring’s Retry Template.

private RetryCallback<Void, TaxWebServiceException> doWebserviceCallWithRetryCallback(TaxCalculation taxCalculation) {
                return new RetryCallback<Void, TaxWebServiceException>() {
                    @Override
                    public Void doWithRetry(RetryContext context) throws TaxWebServiceException {
                        taxPaymentWebService.doWebserviceCallToTaxService(taxCalculation.getEmployee(), taxCalculation.getTax());
                        return null;
                    }
                };
            }
        }

Handling exceptions in the Step configuration

The second step in our job is calling the Tax Payment Webservice. If that succeeds, generating and sending the employees paycheck:

protected Step wsCallAndGenerateAndSendPaycheckStep(String stepName) {
        CompositeItemProcessor<TaxCalculation, PayCheck> compositeItemProcessor = new CompositeItemProcessor<>();
        compositeItemProcessor.setDelegates(Arrays.asList(
                callWebserviceProcessor,
                sendPaycheckProcessor
        ));

        return stepBuilders.get(stepName)
                .<TaxCalculation, PayCheck>chunk(5)
                .faultTolerant()
                .skipPolicy(maxConsecutiveExceptionsSkipPolicy)
                .noRollback(TaxWebServiceNonFatalException.class)
                .noRollback(EmailSenderException.class)
                .reader(wsCallItemReader)
                .processor(compositeItemProcessor)
                .writer(wsCallItemWriter)
                .listener(maxConsecutiveExceptionsSkipPolicy)
                .listener(failedStepStepExecutionListener)
                .allowStartIfComplete(true)
                .build();
    }

We can already handle some failures by the backend thanks to the retry template described above. But what if the Tax Payment Webservice backend keeps providing errors? If the Tax Payment Webservice fails for 5 consecutive employees, something is wrong and the best thing to do is to stop.

Let’s go through the code above step by step:

• First, we need to make our StepBuilder faultTolerant which allows us to specify a skipPolicy and noRollback exceptions
• Second, we need to tell Spring Batch that it should not rollback the chunk for the TaxWebServiceNonFatalException and EmailSenderException exceptions. If one of those exception occurs, we don’t care. The next time we run our Job Instance, these items will be retried and hopefully succeed. If we would not have specified the noRollback exceptions, Spring Batch would restart the complete chunk and omit the item that failed.
• Next, we implement the MaxConsecutiveExceptionsSkipPolicy. In this SkipPolicy, we simply count the TaxWebServiceNonFatalException and EmailSenderException exceptions that occur after each other. Each time the taxes are paid successfully for a certain employee (which we find out by registering as an ItemProcessListener), we reset the counter. And if it fails 5 times in a row, we throw a SkipLimitExceededException telling Spring to stop the job completely. If we would not have specified the skipPolicy, the job would immediately stop.
• And last but not least, we implement a FailedStepStepExecutionListener. This is an implementation of a StepExecutionListener and makes sure that the Job Instance is marked as failed so that we can rerun it later. This is needed because we marked the exceptions as TaxWebServiceNonFatalException and EmailSenderException as skippable.

This all works really fine but there is one big caveat to remember: you really need to make sure that you implement your ItemReader’s carefully! In our case, we want to restart our Job Instance so we need to return all the TaxCalculation objects for which we didn’t find a Paycheck in the database for a given year and month. Since we are using the JpaPagingItemReader, there is one extra thing to keep in mind: the reader is paged and goes to the database per page. An example makes this more clear:

first chunk: first page; no paychecks in DB available => returns items 1 to 5

second chunk: second page; 5 paychecks in DB (first 5 are not returned) => returns items 11 to 15

This means we skipped items 6 to 10! To solve this we need to track the jobExecutionId and select all Tax Calculation records for which we didn’t find a Paycheck record in the database for a given year and month and not the current Job Execution. This all sounds more difficult than it is: just check out the code and take a look at our wsCallAndGenerateAndSendPaycheckStep.

The post Spring Batch: Handling exceptions and retrying appeared first on Cegeka Blog.