All Questions
Tagged with spring-kafka spring-batch
33 questions
0
votes
0
answers
20
views
Remote Partitioning in Spring Batch - Job completion for small number of records can't de done until Job completes for large number of records
I am trying Spring Batch with remote partitioning [ master-salve approach].
I have one master step which sends records to worker nodes via KAFKA.
All was working fine until parallel job executions ...
0
votes
1
answer
25
views
Why doesn't KafkaItemReader support auto-rebalance? How to implement it?
I am working with Spring Batch and using KafkaItemReader to read data from Kafka topics. However, I noticed that KafkaItemReader requires explicitly providing the list of topic partitions during ...
- 1
0
votes
0
answers
45
views
Multi-Threading KafkaItemReader using SynchronizedItemStreamReader in Spring Batch
I'm comparing multiple options regarding to KafkaConsumer in SpringBatch.
With the example from spring batch project, I wrote simple code which runs KafkaItemReader as it is intended to use - single ...
0
votes
1
answer
98
views
Scaling Spring Batch remote partitioning with kafka
I need to scale spring batch job by remote partitioning with kafka.
I want to run manager step on one VM and worker steps on different VMs. For that I have some questions -
If my manager VM instance ...
- 3
0
votes
2
answers
250
views
Save data to Cassandra from Kafka using Spring Batch with high throughput
I need to consume data from Kafka and store it in Cassandra DB using Spring Batch. Let's say that the data is an object with a String id Primary key and int status code. The volume of messages is in ...
0
votes
1
answer
98
views
Spring batch remote partitioning with kafka multitenant
I have implemented the spring batch remote partitioning with Kafka where 3 workers are running on separate JVMs , on the each worker before initialization of job repository I need to set the tenant.
I ...
- 3
2
votes
0
answers
313
views
Massive performance downgrade in Spring Batch application after update the Spring Boot Dependencies version from 2.4.2 to 2.7.6
We have upgraded a Spring Batch application in which the Spring Boot Dependencies is upgrade from version 2.4.2 to 2.7.6. Unfortunately we noticed a performance downgrade in processing the same file ...
- 21
0
votes
1
answer
711
views
Spring Batch remote-partitioning worker-job not reading data from middleware
I wanted to checkout the remote partitioning setup in Spring Batch. I am using Spring Boot v2.7.2 & Kafka as middleware. I have used the ColumnRangePartitioner from the Spring Batch samples
I am ...
- 352
0
votes
1
answer
405
views
Spring Batch: KafkaItemReader reads from offset 0 if the JVM is not restarted
I'm trying to implement a Spring Batch job that is scheduled by a Spring scheduler.
The current befavior is as follows:
The job is launched by the scheduler using JarLauncher
The job consumes the ...
- 495
0
votes
1
answer
806
views
Asynchronous Kafka consumer in Spring Batch Application
In our Spring Batch application workers, item processors are further interacting with another service asynchronously through Kafka. The requirement here is we required an acknowledgement in order to ...
- 27
1
vote
1
answer
1k
views
org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel- occures before subscriber bean is created
I am trying to implement ReplyingKafkaTemplate with spring batch integration.
It is actually running eagerly and throwing the "Dispatcher has no subscribers for channel" error before the ...
- 289
1
vote
1
answer
202
views
Spring batch integration using OutBoundGateway and ReplyingKafkaTemplate
My Goal
I need to read a file and divide each line as a message and send to kafka from a spring batch project and another spring integration project will be receiving the messages to process it in a ...
- 289
1
vote
1
answer
526
views
Intercept Kafka rollback in a Spring Batch
In a Spring Batch I developed an ItemWriter that use a StreamBridge to write to Kafka with a transactional producer.
With a CompositeItemWriter, the step use also a MongoItemWriter, so the output is ...
- 45
1
vote
0
answers
553
views
Controlling Kafka-Offsets using KafkaItemReader and writing to a Database
I'm using the following libraries:
spring-boot 2.5.8
spring-kafka 2.7.9
spring-batch-infrastructure 4.3.4 (including KafkaItemReader and JdbcBatchItemWriter)
I defined a job that reads items from a ...
- 512
0
votes
2
answers
1k
views
Get message headers in batch mode
In a spring-cloud-stream Kafka application, when consuming messages in regular mode I get the Message with its payload and custom headers properly. yet, when I set the consumer "headerMode" ...
