javasetup

CSV File to Oracle Database


After setting up a spring batch application in previous example, we are ready to dive in deep to explore more functionalities that Spring Batch provides.
In this example , we will take a csv file , read it , and write the content into Oracle database.


Example description : We have a csv file USER_DATA_20180930.csv . Using spring batch we will load the content to oracle database table.

CSV File

USER_DATA_20180930.csv

	User_Name,Sex,Phone_No,Address
	Pia,F,11111111,Test Address 1
	Sarat,M,222222222,Test Address 2
	Zaheer,M,333333333,Test Address 3
	Arthur,M,444444444,Test Address 4
	




In this section we will not use any build tools like Maven/Gradle instead we will manually add all the dependency jars into classpath
and let our IDE do the project building.



1. Prerequisites

Before we start coding let us first tell you the tools we used and the libraries we need for spring batch development.

Tools Used
  Eclipse    [Any version 4.2(Juno) or above . We are using 4.8(Photon).]

JDK version
  JDK 1.6     [We are using JDK 1.6 to build this application.]

Jars Required
  We need following dependency jars to build the application. [Download link provided within]


Download all jars


Now we are done with all the things we need to build a basic spring batch application. Lets get started then. Open eclipse and follow below steps .

2. Create a Java project


Of course you know how to do it, but we will not skip any step .

Follow the steps , read inline instructions.

Step 1
Spring Batch
Goto File --> New --> Java Project

Step 2
Spring Batch
Give a name to your application and click on Finish button.

	
Now we have created a Java Projet in eclipse. Next step is to add the downloaded jars into the build path of this project.

3. Configure Build Path


Steps to configure build path in eclipse. Read inline instructions.

Step 1
Spring Batch
Right-Click on the project
Goto Build Path
Click on Configure Build Path

Step 2
Spring Batch
Click on Add Exsternal Jars... button

Step 3
Spring Batch
Navigate to the directory where the jars has been downloaded.
Select all the downloaded jars and click on Open.

Step 4
Spring Batch
Click on apply and close button.

	
Now we are done with configuring build path. Lets start coding :)

4. Code


This application contains following files :

1. jobConfiguration.xml - Batch job configuration xml.
2. UserDataBean.java - Bean to hold user details.
3. CustomItemProcessor.java - Processes the data.
4. DemoBatchListener.java - Listner class to implement tasks before or after Job/Step execution(Optional for this example).
5. context.xml - Spring Application context(Defining JobLauncher).
6. config.properties - Property file for configurable parameters.
7. JobController.java - Batch job invoker class.

  • Review Directory structure of the project in below screen. Read inline comments.
Project structure in eclipse
Spring Batch
Package config contains configuration xmls and property file
Package com.javasetup.springbatch.beans contains UserDataBean class
Package com.javasetup.springbatch.contoller contains JobController class
Package com.javasetup.springbatch.listeners contains DemoBatchListener class
Package com.javasetup.springbatch.processors contains CustomItemProcessor class



  • At a minimum, launching a batch job requires two things: the Job to be launched and a JobLauncher.
A Job is an entity that encapsulates an entire batch process. As is common with other Spring projects, a Job will be wired together via an XML configuration file or Java based configuration. This configuration may be referred to as the job configuration.

Lets configuring a job first as shown in jobConfiguration.xml.
  • Provide schema location for Spring Batch and Spring Beans
  • Define a job UserDataLoaderJob which contains a step Step1.
  • Define a tasklet inside Step1.
  • Define a Chunk inside tasklet tag.
  • This chunk contains one reader flatFileItemReader one processor fileProcessor and one writer sqlItemWriter.
  • This chunk also contains commit-interval value ${db.commit.interval} which is defined in config.properties file.
  • commit-interval - The number of items that will be processed before the transaction is committed.
  • Though it is not required for this example but to know how a listener can be attached to a step/chunk ,just define a batch listener batchListener inside the step Step1 as shown in the jobConfiguration.xml.
  • Use import tag to locate context.xml file.
  • Define the beans.
jobConfiguration.xml

<beans xmlns="http://www.springframework.org/schema/beans" 
   xmlns:batch="http://www.springframework.org/schema/batch"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
   xsi:schemaLocation="http://www.springframework.org/schema/batch 
      http://www.springframework.org/schema/batch/spring-batch-2.2.xsd
      http://www.springframework.org/schema/beans
      http://www.springframework.org/schema/beans/spring-beans-3.2.xsd"> 
    
   <import resource="context.xml" />

	<!-- Defining beans -->
	<bean id="userDataBean" class="com.javasetup.springbatch.beans.UserDataBean" scope="prototype" />
    <bean id="fileProcessor" class="com.javasetup.springbatch.processors.CustomItemProcessor" />
    <bean id="batchListener" class="com.javasetup.springbatch.listeners.DemoBatchListener" />
	
	<!-- Defining a job-->
	<batch:job id="UserDataLoaderJob">
		<!-- Defining a Step -->
		<batch:step id="step1">
			<batch:tasklet>
				<batch:chunk reader="flatFileItemReader" processor="fileProcessor"
					writer="sqlItemWriter" commit-interval="${db.commit.interval}">
				</batch:chunk>
			</batch:tasklet>
			<batch:listeners>
			  <batch:listener ref="batchListener" />
			</batch:listeners>
		</batch:step>
	</batch:job>



	<bean id="flatFileItemReader" scope="step" class="org.springframework.batch.item.file.FlatFileItemReader">
	   <property name="resource" value="file:${user.file.location}\\${user.file.pattern}" />
		<property name="lineMapper" ref="csvFileLineMapper" />
	</bean>


	<bean id="csvFileLineMapper" class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
		<property name="lineTokenizer" ref="csvFileLineTokenizer" />
		<property name="fieldSetMapper" ref="fieldSetMapper" />
	</bean>

	<!-- split the line -->
	<bean id="csvFileLineTokenizer"	class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
		<property name="names" value="${user.file.data.pattern}" />
	</bean>

	<!-- map the data to an object -->
	<bean id="fieldSetMapper" class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
		<property name="prototypeBeanName" value="userDataBean" />
	</bean>

    <!-- Writer to Database table-->
  <bean id="sqlItemWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter">
	<property name="dataSource" ref="dataSource" />
	<property name="sql">
	  <value>
            <![CDATA[${user.data.insert.query}]]>
	  </value>
	</property>
	<!-- Maps object property provided in sql query -->
	<property name="itemSqlParameterSourceProvider">
		<bean class="org.springframework.batch.item.database.BeanPropertyItemSqlParameterSourceProvider" />
	</property>
  </bean>
</beans> 
	

  • flatFileItemReader - It has two properties. One is resource that contains the loaction of the file to be read, and the other is the lineMapper which will help to read the file.

    lineMapper - It has two properties, One is to split each line(lineTokenizer) and the other is to map the fields into an object(fieldSetMapper).

  • lineTokenizer - has a property names that contains the data pattern(which is required to split the data). value of ${user.file.data.pattern} is in config.properties file.

    fieldSetMapper - has a property prototypeBeanName , which is of com.javasetup.springbatch.beans.UserDataBean type and after splitting the line data spring batch loads the data into this object.

UserDataBean.java

package com.javasetup.springbatch.beans;

public class UserDataBean {
private String userName;
private String sex;
private String phoneNo;
private String address;
public String getUserName() {
	return userName;
}
public void setUserName(String userName) {
	this.userName = userName;
}
public String getSex() {
	return sex;
}
public void setSex(String sex) {
	this.sex = sex;
}
public String getPhoneNo() {
	return phoneNo;
}
public void setPhoneNo(String phoneNo) {
	this.phoneNo = phoneNo;
}
public String getAddress() {
	return address;
}
public void setAddress(String address) {
	this.address = address;
}
@Override
public String toString() {
	return "UserDataBean [userName=" + userName + ", sex=" + sex + ", phoneNo=" + phoneNo + ", address=" + address
			+ "]";
}
}

	

  • fileProcessor - it is of com.javasetup.springbatch.processors.CustomItemProcessor type . It is to process the data if required. Here we are only printing the data in console. [Note : This example does not require a processor.]
CustomItemProcessor.java

package com.javasetup.springbatch.processors;

import org.springframework.batch.item.ItemProcessor;

import com.javasetup.springbatch.beans.UserDataBean;

public class CustomItemProcessor implements ItemProcessor {

	@Override
	public Object process(Object item) throws Exception {
		System.out.println("Inside Item processor :: "+((UserDataBean)item).toString());
		return item;
	}

}

	

  • sqlItemWriter - it is of org.springframework.batch.item.database.JdbcBatchItemWriter type. It contains three properties . One is dataSource which refers datasource bean(used for database connectivity) defined in context.xml file , one is sql which contains the insert query, and the other one is itemSqlParameterSourceProvider.
    itemSqlParameterSourceProvider - it is of org.springframework.batch.item.database.BeanPropertyItemSqlParameterSourceProvider type, which helps to map between oracle table column and bean(UserDataBean) property during data insertion.

  • batchListener - it is of com.javasetup.springbatch.listeners.DemoBatchListener type. Use to do operation before/after step/chunk execution.
DemoBatchListener.java

package com.javasetup.springbatch.listeners;

import org.springframework.batch.core.ChunkListener;
import org.springframework.batch.core.ExitStatus;
import org.springframework.batch.core.StepExecution;
import org.springframework.batch.core.StepExecutionListener;
import org.springframework.batch.core.scope.context.ChunkContext;

public class DemoBatchListener implements ChunkListener,StepExecutionListener{

	@Override
	public ExitStatus afterStep(StepExecution step) {
		System.out.println("After Step execution :: "+step.getSummary());
		return null;
	}

	@Override
	public void beforeStep(StepExecution step) {
		System.out.println("Before Step execution :: "+step.getSummary());
		
	}

	@Override
	public void afterChunk(ChunkContext chunk) {
		//System.out.println("After chunk execution :: "+chunk.getStepContext().getStepExecution().getSummary());
		
	}

	@Override
	public void afterChunkError(ChunkContext arg0) {
		
	}

	@Override
	public void beforeChunk(ChunkContext chunk) {
		//System.out.println("Before chunk execution :: "+chunk.getStepContext().getStepExecution().getSummary());
		
	}

}

	



Next step is to create a JobLauncher that executes the created Job(UserDataLoaderJob).
  • A JobLauncher represents a simple interface for launching a Job with a given set of JobParameters.
  • The most basic implementation of the JobLauncher interface is the SimpleJobLauncher. Its only required dependency is a JobRepository, in order to obtain an execution.
  • Configure spring context to define jobLauncher(a JobLauncher), jobRepository and transactioManager bean as shown in the context.xml below.
  • Also define a bean datasource for database connection. Provide the driver(like driver class name) and database details(like connection url , username and password).
  • ${db.host.url}, ${db.port.number}, ${db.service.name}, ${db.user}, ${db.password} are defineed in config.properties file.
  • Provide location of config.properties file in context:property-placeholder tag.
context.xml

     <beans xmlns="http://www.springframework.org/schema/beans" 
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
   xmlns:context="http://www.springframework.org/schema/context"
   xsi:schemaLocation="http://www.springframework.org/schema/beans 
      http://www.springframework.org/schema/beans/spring-beans-3.2.xsd 
      http://www.springframework.org/schema/context
      http://www.springframework.org/schema/context/spring-context-3.2.xsd">  
   
     <context:property-placeholder location="classpath:/config/config.properties" />
     
   <bean id="jobRepository"   
      class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"> 
      <property name="transactionManager" ref="transactionManager" /> 
   </bean>     
     
     <!-- connect to database -->
  <bean id="dataSource"
	class="org.springframework.jdbc.datasource.DriverManagerDataSource">
	<property name="driverClassName" value="oracle.jdbc.driver.OracleDriver" />
	<property name="url" value="jdbc:oracle:thin:@//${db.host.url}:${db.port.number}/${db.service.name}" />
	<property name="username" value="${db.user}" />
	<property name="password" value="${db.password}" />
  </bean>
    
   <bean id="transactionManager" 
      class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />  
      
   <bean id="jobLauncher" 
      class="org.springframework.batch.core.launch.support.SimpleJobLauncher"> 
      <property name="jobRepository" ref="jobRepository" /> 
   </bean> 

   
</beans> 
	


The configuration properties file.
config.properties

###########################################################################
######            Java Setup Batch Process Config Property           ######
###########################################################################

############# DB Details ############
db.host.url=127.0.0.1
db.port.number=1521
db.service.name=xe
db.user=JAVA_SETUP
db.password=JAVA_SETUP
db.commit.interval=2
############# App Specific properties #############
user.file.location=D:\\input\\csv
user.file.pattern=USER_DATA_20180930.csv
user.file.data.pattern=User_Name,Sex,Phone_No,Address
user.data.insert.query=Insert into USER_DETAILS (USER_NAME,SEX,PHONE_NO,ADDRESS) values (:userName,:sex,:phoneNo,:address)
####################################################
	


5. Run the Job

  • Create the job launcher.
  • Create the job.
  • Execute the job using run method of jobLauncher.
JobController.java

package com.javasetup.springbatch.contoller;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

public class JobController {

	public static void main(String[] args) throws Exception {
		  System.out.println("Inside Spring bactch Job controller [Start]"); 
	      String[] springConfig  =  {"config/jobConfiguration.xml"};  
	      
	      // Creating the application context object  
	      ApplicationContext context = new ClassPathXmlApplicationContext(springConfig); 
	      
	      // Creating the job launcher 
	      JobLauncher jobLauncher = (JobLauncher) context.getBean("jobLauncher"); 
	  
	      // Creating the job 
	      Job job = (Job) context.getBean("UserDataLoaderJob"); 
	  
	      // Executing the JOB 
	      JobExecution execution = jobLauncher.run(job, new JobParameters()); 
	      System.out.println("Excution Status : " + execution.getStatus());
	      System.out.println("Inside Spring bactch Job controller [End]");
	}

}
    
	


  • To execute the application , follow below step.
Execute Application
Spring Batch
Right-Click on the JobController.java
Select Run As -> Java Aplication


The console output that we got :
Console Output

Inside Spring bactch Job controller [Start]
18 Oct, 2018 12:52:44 PM org.springframework.context.support.ClassPathXmlApplicationContext prepareRefresh
INFO: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@1113708: startup date [Thu Oct 18 12:52:44 IST 2018]; root of context hierarchy
18 Oct, 2018 12:52:44 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [config/jobConfiguration.xml]
18 Oct, 2018 12:52:44 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [config/context.xml]
18 Oct, 2018 12:52:44 PM org.springframework.beans.factory.support.DefaultListableBeanFactory registerBeanDefinition
INFO: Overriding bean definition for bean 'UserDataLoaderJob': replacing [Generic bean: class [org.springframework.batch.core.configuration.xml.SimpleFlowFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] with [Generic bean: class [org.springframework.batch.core.configuration.xml.JobParserJobFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
18 Oct, 2018 12:52:44 PM org.springframework.context.support.PropertySourcesPlaceholderConfigurer loadProperties
INFO: Loading properties file from class path resource [config/config.properties]
18 Oct, 2018 12:52:44 PM org.springframework.beans.factory.support.DefaultListableBeanFactory registerBeanDefinition
INFO: Overriding bean definition for bean 'flatFileItemReader': replacing [Generic bean: class [org.springframework.batch.item.file.FlatFileItemReader]; scope=step; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=false; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in class path resource [config/jobConfiguration.xml]] with [Root bean: class [org.springframework.aop.scope.ScopedProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in BeanDefinition defined in class path resource [config/jobConfiguration.xml]]
18 Oct, 2018 12:52:44 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@17725c4: defining beans [org.springframework.context.support.PropertySourcesPlaceholderConfigurer#0,jobRepository,dataSource,transactionManager,jobLauncher,userDataBean,fileProcessor,batchListener,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,step1,UserDataLoaderJob,flatFileItemReader,csvFileLineMapper,csvFileLineTokenizer,fieldSetMapper,sqlItemWriter,scopedTarget.flatFileItemReader]; root of factory hierarchy
18 Oct, 2018 12:52:44 PM org.springframework.jdbc.datasource.DriverManagerDataSource setDriverClassName
INFO: Loaded JDBC driver: oracle.jdbc.driver.OracleDriver
18 Oct, 2018 12:52:44 PM org.springframework.batch.core.launch.support.SimpleJobLauncher afterPropertiesSet
INFO: No TaskExecutor has been set, defaulting to synchronous executor.
18 Oct, 2018 12:52:45 PM org.springframework.batch.core.launch.support.SimpleJobLauncher run
INFO: Job: [FlowJob: [name=UserDataLoaderJob]] launched with the following parameters: [{}]
18 Oct, 2018 12:52:45 PM org.springframework.batch.core.job.SimpleStepHandler handleStep
INFO: Executing step: [step1]
Before Step execution :: StepExecution: id=1, version=1, name=step1, status=STARTED, exitStatus=EXECUTING, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=0, rollbackCount=0
Inside Item processor :: UserDataBean [userName=User_Name, sex=Sex, phoneNo=Phone_No, address=Address]
Inside Item processor :: UserDataBean [userName=Pia, sex=F, phoneNo=11111111, address=Test Address 1]
Inside Item processor :: UserDataBean [userName=Sarat, sex=M, phoneNo=222222222, address=Test Address 2]
Inside Item processor :: UserDataBean [userName=Zaheer, sex=M, phoneNo=333333333, address=Test Address 3]
Inside Item processor :: UserDataBean [userName=Arthur, sex=M, phoneNo=444444444, address=Test Address 4]
After Step execution :: StepExecution: id=1, version=4, name=step1, status=COMPLETED, exitStatus=COMPLETED, readCount=5, filterCount=0, writeCount=5 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=3, rollbackCount=0
18 Oct, 2018 12:52:45 PM org.springframework.batch.core.launch.support.SimpleJobLauncher run
INFO: Job: [FlowJob: [name=UserDataLoaderJob]] completed with the following parameters: [{}] and the following status: [COMPLETED]
Excution Status : COMPLETED
Inside Spring bactch Job controller [End]
    
	


The content of USER_DATA_20180930.csv file is inserted into database table USER_DETAILS.
USER_DETAILS
Spring Batch

In this section we will use Maven to build the project.



1. Prerequisites

Before we start coding let us first tell you the tools we used and the libraries we need for spring batch development.

Tools Used
  Eclipse    [Any version 4.2(Juno) or above . We are using 4.8(Photon).]
  Maven     [Plugin version 3.1 ]

JDK & Libraries
  JDK 1.8     [We are using JDK 1.8 to build this application.]
  Spring Core 3.2.2.RELEASE
  Spring Jdbc 3.2.2.RELEASE
  Spring Batch 2.2.0.RELEASE
  Ojdbc6 11.2.0.3

    Thats all. Now we can start developing the application.
	

2. Create a Maven project


Of course you know how to do it, but we will not skip any step .

Follow the steps , read inline instructions.

Step 1
Spring Batch
Goto File --> New --> Other


Step 2
Spring Batch
Maven --> Maven Project . Click on next button.


Step 3
Spring Batch
Check Create a simple project as shown.
Check Use default Workspace location as shown.
Click on next button.


Step 4
Spring Batch
Write Group Id.
Write Artifact Id(jar name).
Write Name.
Click on Finish button.


    With that step 4 we are done with creating a Maven Projet in eclipse.
	


  • Review Directory structure of the project just after creating it in below screen.
Project structure in eclipse
Spring Batch


3. Configure Build Path

  • We will configure the build path using maven.
  • To do that we need to mention dependencies in pom.xml.
  • This application needs only these two dependencies
       1. Spring Core 3.2.2.RELEASE.
       2. Spring Jdbc 3.2.2.RELEASE.
       3. Spring Batch 2.2.0.RELEASE.
       4. Ojdbc6 11.2.0.3.
  • Content of pom.xml shared below
pom.xml

  <project xmlns="http://maven.apache.org/POM/4.0.0"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.javasetup.springbatch</groupId>
	<artifactId>SpringBatchMavenExample1</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<name>SpringBatchMavenExample1</name>

	<properties>
		<spring.version>3.2.2.RELEASE</spring.version>
		<spring.batch.version>2.2.0.RELEASE</spring.batch.version>
	</properties>

	<dependencies>

		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-core</artifactId>
			<version>${spring.version}</version>
		</dependency>
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-jdbc</artifactId>
			<version>${spring.version}</version>
		</dependency>
		<dependency>
			<groupId>org.springframework.batch</groupId>
			<artifactId>spring-batch-core</artifactId>
			<version>${spring.batch.version}</version>
		</dependency>

	<dependency>
		<groupId>com.oracle</groupId>
		<artifactId>ojdbc</artifactId>
		<version>11.2.0</version>
        <scope>system</scope>
        <systemPath>....../SpringBatchMavenExample1/lib/ojdbc6.jar</systemPath>
	</dependency>

	</dependencies>

</project>
	


  • Follow below step to add dependencies into build path.
Step 1
Spring Batch
Right-Click on the project.
Go to Maven -> Update Project...

Step 2
Spring Batch
Click OK in the Update Maven Project pop-up.


  • Below jars will get downloaded and addded to the classpath automatically by Maven.
Maven Dependencies
Spring Batch
	
Now we are done with configuring build path. Lets start coding :)

4. Code


This application contains following files :

1. pom.xml - Maven config file.
2. jobConfiguration.xml - Batch job configuration xml.
3. UserDataBean.java - Bean to hold user details.
4. CustomItemProcessor.java - Processes the data.
5. DemoBatchListener.java - Listner class to implement tasks before or after Job/Step execution(Optional for this example).
6. context.xml - Spring Application context(Defining JobLauncher).
7. config.properties - Property file for configurable parameters.
8. JobController.java - Batch job invoker class.

  • Review Directory structure of the project in below screen. Read inline comments.
Project structure in eclipse
Spring Batch
Package config contains configuration xmls and property file
Package com.javasetup.springbatch.beans contains UserDataBean class
Package com.javasetup.springbatch.contoller contains JobController class
Package com.javasetup.springbatch.listeners contains DemoBatchListener class
Package com.javasetup.springbatch.processors contains CustomItemProcessor class

  • We hve already talked about pom.xml file. Find the pom.xml content below.
pom.xml

  <project xmlns="http://maven.apache.org/POM/4.0.0"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.javasetup.springbatch</groupId>
	<artifactId>SpringBatchMavenExample1</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<name>SpringBatchMavenExample1</name>

	<properties>
		<spring.version>3.2.2.RELEASE</spring.version>
		<spring.batch.version>2.2.0.RELEASE</spring.batch.version>
	</properties>

	<dependencies>

		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-core</artifactId>
			<version>${spring.version}</version>
		</dependency>
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-jdbc</artifactId>
			<version>${spring.version}</version>
		</dependency>
		<dependency>
			<groupId>org.springframework.batch</groupId>
			<artifactId>spring-batch-core</artifactId>
			<version>${spring.batch.version}</version>
		</dependency>

	<dependency>
		<groupId>com.oracle</groupId>
		<artifactId>ojdbc</artifactId>
		<version>11.2.0</version>
        <scope>system</scope>
        <systemPath>....../SpringBatchMavenExample1/lib/ojdbc6.jar</systemPath>
	</dependency>

	</dependencies>

</project>
	

You may face issue while loading ojdbc6 jar using pom.xml . To understand and resolve that read this link : Missing artifact com.oracle:ojdbc6:jar:11.2.0.3


  • At a minimum, launching a batch job requires two things: the Job to be launched and a JobLauncher.
A Job is an entity that encapsulates an entire batch process. As is common with other Spring projects, a Job will be wired together via an XML configuration file or Java based configuration. This configuration may be referred to as the job configuration.

Lets configuring a job first as shown in jobConfiguration.xml.
  • Provide schema location for Spring Batch and Spring Beans
  • Define a job UserDataLoaderJob which contains a step Step1.
  • Define a tasklet inside Step1.
  • Define a Chunk inside tasklet tag.
  • This chunk contains one reader flatFileItemReader one processor fileProcessor and one writer sqlItemWriter.
  • This chunk also contains commit-interval value ${db.commit.interval} which is defined in config.properties file.
  • commit-interval - The number of items that will be processed before the transaction is committed.
  • Though it is not required for this example but to know how a listener can be attached to a step/chunk ,just define a batch listener batchListener inside the step Step1 as shown in the jobConfiguration.xml.
  • Use import tag to locate context.xml file.
  • Define the beans.
jobConfiguration.xml

<beans xmlns="http://www.springframework.org/schema/beans" 
   xmlns:batch="http://www.springframework.org/schema/batch"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
   xsi:schemaLocation="http://www.springframework.org/schema/batch 
      http://www.springframework.org/schema/batch/spring-batch-2.2.xsd
      http://www.springframework.org/schema/beans
      http://www.springframework.org/schema/beans/spring-beans-3.2.xsd"> 
    
   <import resource="context.xml" />

	<!-- Defining beans -->
	<bean id="userDataBean" class="com.javasetup.springbatch.beans.UserDataBean" scope="prototype" />
    <bean id="fileProcessor" class="com.javasetup.springbatch.processors.CustomItemProcessor" />
    <bean id="batchListener" class="com.javasetup.springbatch.listeners.DemoBatchListener" />
	
	<!-- Defining a job-->
	<batch:job id="UserDataLoaderJob">
		<!-- Defining a Step -->
		<batch:step id="step1">
			<batch:tasklet>
				<batch:chunk reader="flatFileItemReader" processor="fileProcessor"
					writer="sqlItemWriter" commit-interval="${db.commit.interval}">
				</batch:chunk>
			</batch:tasklet>
			<batch:listeners>
			  <batch:listener ref="batchListener" />
			</batch:listeners>
		</batch:step>
	</batch:job>



	<bean id="flatFileItemReader" scope="step" class="org.springframework.batch.item.file.FlatFileItemReader">
	   <property name="resource" value="file:${user.file.location}\\${user.file.pattern}" />
		<property name="lineMapper" ref="csvFileLineMapper" />
	</bean>


	<bean id="csvFileLineMapper" class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
		<property name="lineTokenizer" ref="csvFileLineTokenizer" />
		<property name="fieldSetMapper" ref="fieldSetMapper" />
	</bean>

	<!-- split the line -->
	<bean id="csvFileLineTokenizer"	class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
		<property name="names" value="${user.file.data.pattern}" />
	</bean>

	<!-- map the data to an object -->
	<bean id="fieldSetMapper" class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
		<property name="prototypeBeanName" value="userDataBean" />
	</bean>

    <!-- Writer to Database table-->
  <bean id="sqlItemWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter">
	<property name="dataSource" ref="dataSource" />
	<property name="sql">
	  <value>
            <![CDATA[${user.data.insert.query}]]>
	  </value>
	</property>
	<!-- Maps object property provided in sql query -->
	<property name="itemSqlParameterSourceProvider">
		<bean class="org.springframework.batch.item.database.BeanPropertyItemSqlParameterSourceProvider" />
	</property>
  </bean>
</beans> 
	

  • flatFileItemReader - It has two properties. One is resource that contains the loaction of the file to be read, and the other is the lineMapper which will help to read the file.

    lineMapper - It has two properties, One is to split each line(lineTokenizer) and the other is to map the fields into an object(fieldSetMapper).

  • lineTokenizer - has a property names that contains the data pattern(which is required to split the data). value of ${user.file.data.pattern} is in config.properties file.

    fieldSetMapper - has a property prototypeBeanName , which is of com.javasetup.springbatch.beans.UserDataBean type and after splitting the line data spring batch loads the data into this object.

UserDataBean.java

package com.javasetup.springbatch.beans;

public class UserDataBean {
private String userName;
private String sex;
private String phoneNo;
private String address;
public String getUserName() {
	return userName;
}
public void setUserName(String userName) {
	this.userName = userName;
}
public String getSex() {
	return sex;
}
public void setSex(String sex) {
	this.sex = sex;
}
public String getPhoneNo() {
	return phoneNo;
}
public void setPhoneNo(String phoneNo) {
	this.phoneNo = phoneNo;
}
public String getAddress() {
	return address;
}
public void setAddress(String address) {
	this.address = address;
}
@Override
public String toString() {
	return "UserDataBean [userName=" + userName + ", sex=" + sex + ", phoneNo=" + phoneNo + ", address=" + address
			+ "]";
}
}

	

  • fileProcessor - it is of com.javasetup.springbatch.processors.CustomItemProcessor type . It is to process the data if required. Here we are only printing the data in console. [Note : This example does not require a processor.]
CustomItemProcessor.java

package com.javasetup.springbatch.processors;

import org.springframework.batch.item.ItemProcessor;

import com.javasetup.springbatch.beans.UserDataBean;

public class CustomItemProcessor implements ItemProcessor {

	@Override
	public Object process(Object item) throws Exception {
		System.out.println("Inside Item processor :: "+((UserDataBean)item).toString());
		return item;
	}

}

	

  • sqlItemWriter - it is of org.springframework.batch.item.database.JdbcBatchItemWriter type. It contains three properties . One is dataSource which refers datasource bean(used for database connectivity) defined in context.xml file , one is sql which contains the insert query, and the other one is itemSqlParameterSourceProvider.
    itemSqlParameterSourceProvider - it is of org.springframework.batch.item.database.BeanPropertyItemSqlParameterSourceProvider type, which helps to map between oracle table column and bean(UserDataBean) property during data insertion.

  • batchListener - it is of com.javasetup.springbatch.listeners.DemoBatchListener type. Use to do operation before/after step/chunk execution.
DemoBatchListener.java

package com.javasetup.springbatch.listeners;

import org.springframework.batch.core.ChunkListener;
import org.springframework.batch.core.ExitStatus;
import org.springframework.batch.core.StepExecution;
import org.springframework.batch.core.StepExecutionListener;
import org.springframework.batch.core.scope.context.ChunkContext;

public class DemoBatchListener implements ChunkListener,StepExecutionListener{

	@Override
	public ExitStatus afterStep(StepExecution step) {
		System.out.println("After Step execution :: "+step.getSummary());
		return null;
	}

	@Override
	public void beforeStep(StepExecution step) {
		System.out.println("Before Step execution :: "+step.getSummary());
		
	}

	@Override
	public void afterChunk(ChunkContext chunk) {
		//System.out.println("After chunk execution :: "+chunk.getStepContext().getStepExecution().getSummary());
		
	}

	@Override
	public void afterChunkError(ChunkContext arg0) {
		
	}

	@Override
	public void beforeChunk(ChunkContext chunk) {
		//System.out.println("Before chunk execution :: "+chunk.getStepContext().getStepExecution().getSummary());
		
	}

}

	



Next step is to create a JobLauncher that executes the created Job(UserDataLoaderJob).
  • A JobLauncher represents a simple interface for launching a Job with a given set of JobParameters.
  • The most basic implementation of the JobLauncher interface is the SimpleJobLauncher. Its only required dependency is a JobRepository, in order to obtain an execution.
  • Configure spring context to define jobLauncher(a JobLauncher), jobRepository and transactioManager bean as shown in the context.xml below.
  • Also define a bean datasource for database connection. Provide the driver(like driver class name) and database details(like connection url , username and password).
  • ${db.host.url}, ${db.port.number}, ${db.service.name}, ${db.user}, ${db.password} are defineed in config.properties file.
  • Provide location of config.properties file in context:property-placeholder tag.
context.xml

     <beans xmlns="http://www.springframework.org/schema/beans" 
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
   xmlns:context="http://www.springframework.org/schema/context"
   xsi:schemaLocation="http://www.springframework.org/schema/beans 
      http://www.springframework.org/schema/beans/spring-beans-3.2.xsd 
      http://www.springframework.org/schema/context
      http://www.springframework.org/schema/context/spring-context-3.2.xsd">  
   
     <context:property-placeholder location="classpath:/config/config.properties" />
     
   <bean id="jobRepository"   
      class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"> 
      <property name="transactionManager" ref="transactionManager" /> 
   </bean>     
     
     <!-- connect to database -->
  <bean id="dataSource"
	class="org.springframework.jdbc.datasource.DriverManagerDataSource">
	<property name="driverClassName" value="oracle.jdbc.driver.OracleDriver" />
	<property name="url" value="jdbc:oracle:thin:@//${db.host.url}:${db.port.number}/${db.service.name}" />
	<property name="username" value="${db.user}" />
	<property name="password" value="${db.password}" />
  </bean>
    
   <bean id="transactionManager" 
      class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />  
      
   <bean id="jobLauncher" 
      class="org.springframework.batch.core.launch.support.SimpleJobLauncher"> 
      <property name="jobRepository" ref="jobRepository" /> 
   </bean> 

   
</beans> 
	


The configuration properties file.
config.properties

###########################################################################
######            Java Setup Batch Process Config Property           ######
###########################################################################

############# DB Details ############
db.host.url=127.0.0.1
db.port.number=1521
db.service.name=xe
db.user=JAVA_SETUP
db.password=JAVA_SETUP
db.commit.interval=2
############# App Specific properties #############
user.file.location=D:\\input\\csv
user.file.pattern=USER_DATA_20180930.csv
user.file.data.pattern=User_Name,Sex,Phone_No,Address
user.data.insert.query=Insert into USER_DETAILS (USER_NAME,SEX,PHONE_NO,ADDRESS) values (:userName,:sex,:phoneNo,:address)
####################################################
	


5. Run the Job

  • Create the job launcher.
  • Create the job.
  • Execute the job using run method of jobLauncher.
JobController.java

package com.javasetup.springbatch.contoller;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

public class JobController {

	public static void main(String[] args) throws Exception {
		  System.out.println("Inside Spring bactch Job controller [Start]"); 
	      String[] springConfig  =  {"config/jobConfiguration.xml"};  
	      
	      // Creating the application context object  
	      ApplicationContext context = new ClassPathXmlApplicationContext(springConfig); 
	      
	      // Creating the job launcher 
	      JobLauncher jobLauncher = (JobLauncher) context.getBean("jobLauncher"); 
	  
	      // Creating the job 
	      Job job = (Job) context.getBean("UserDataLoaderJob"); 
	  
	      // Executing the JOB 
	      JobExecution execution = jobLauncher.run(job, new JobParameters()); 
	      System.out.println("Excution Status : " + execution.getStatus());
	      System.out.println("Inside Spring bactch Job controller [End]");
	}

}
    
	


  • To execute the application , follow below step.
Execute Application
Spring Batch
Right-Click on the JobController.java
Select Run As -> Java Aplication


The console output that we got :
Console Output

Inside Spring bactch Job controller [Start]
18 Oct, 2018 12:52:44 PM org.springframework.context.support.ClassPathXmlApplicationContext prepareRefresh
INFO: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@1113708: startup date [Thu Oct 18 12:52:44 IST 2018]; root of context hierarchy
18 Oct, 2018 12:52:44 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [config/jobConfiguration.xml]
18 Oct, 2018 12:52:44 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [config/context.xml]
18 Oct, 2018 12:52:44 PM org.springframework.beans.factory.support.DefaultListableBeanFactory registerBeanDefinition
INFO: Overriding bean definition for bean 'UserDataLoaderJob': replacing [Generic bean: class [org.springframework.batch.core.configuration.xml.SimpleFlowFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] with [Generic bean: class [org.springframework.batch.core.configuration.xml.JobParserJobFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
18 Oct, 2018 12:52:44 PM org.springframework.context.support.PropertySourcesPlaceholderConfigurer loadProperties
INFO: Loading properties file from class path resource [config/config.properties]
18 Oct, 2018 12:52:44 PM org.springframework.beans.factory.support.DefaultListableBeanFactory registerBeanDefinition
INFO: Overriding bean definition for bean 'flatFileItemReader': replacing [Generic bean: class [org.springframework.batch.item.file.FlatFileItemReader]; scope=step; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=false; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in class path resource [config/jobConfiguration.xml]] with [Root bean: class [org.springframework.aop.scope.ScopedProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in BeanDefinition defined in class path resource [config/jobConfiguration.xml]]
18 Oct, 2018 12:52:44 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@17725c4: defining beans [org.springframework.context.support.PropertySourcesPlaceholderConfigurer#0,jobRepository,dataSource,transactionManager,jobLauncher,userDataBean,fileProcessor,batchListener,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,step1,UserDataLoaderJob,flatFileItemReader,csvFileLineMapper,csvFileLineTokenizer,fieldSetMapper,sqlItemWriter,scopedTarget.flatFileItemReader]; root of factory hierarchy
18 Oct, 2018 12:52:44 PM org.springframework.jdbc.datasource.DriverManagerDataSource setDriverClassName
INFO: Loaded JDBC driver: oracle.jdbc.driver.OracleDriver
18 Oct, 2018 12:52:44 PM org.springframework.batch.core.launch.support.SimpleJobLauncher afterPropertiesSet
INFO: No TaskExecutor has been set, defaulting to synchronous executor.
18 Oct, 2018 12:52:45 PM org.springframework.batch.core.launch.support.SimpleJobLauncher run
INFO: Job: [FlowJob: [name=UserDataLoaderJob]] launched with the following parameters: [{}]
18 Oct, 2018 12:52:45 PM org.springframework.batch.core.job.SimpleStepHandler handleStep
INFO: Executing step: [step1]
Before Step execution :: StepExecution: id=1, version=1, name=step1, status=STARTED, exitStatus=EXECUTING, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=0, rollbackCount=0
Inside Item processor :: UserDataBean [userName=User_Name, sex=Sex, phoneNo=Phone_No, address=Address]
Inside Item processor :: UserDataBean [userName=Pia, sex=F, phoneNo=11111111, address=Test Address 1]
Inside Item processor :: UserDataBean [userName=Sarat, sex=M, phoneNo=222222222, address=Test Address 2]
Inside Item processor :: UserDataBean [userName=Zaheer, sex=M, phoneNo=333333333, address=Test Address 3]
Inside Item processor :: UserDataBean [userName=Arthur, sex=M, phoneNo=444444444, address=Test Address 4]
After Step execution :: StepExecution: id=1, version=4, name=step1, status=COMPLETED, exitStatus=COMPLETED, readCount=5, filterCount=0, writeCount=5 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=3, rollbackCount=0
18 Oct, 2018 12:52:45 PM org.springframework.batch.core.launch.support.SimpleJobLauncher run
INFO: Job: [FlowJob: [name=UserDataLoaderJob]] completed with the following parameters: [{}] and the following status: [COMPLETED]
Excution Status : COMPLETED
Inside Spring bactch Job controller [End]
    
	


The content of USER_DATA_20180930.csv file is inserted into database table USER_DETAILS.
USER_DETAILS
Spring Batch


References