Replace Pioneer Home   All Examples   Free Download

 New request --free  RSS: Replace Pioneer Examples

915.Replace text in multiple files -- How to remove duplicate sentences across many articles in different files?

User: editor -- 2012-02-07          << 914  916 >>
Hits: 3661
Type: Replace text in multiple files   
Search all Replace text in multiple files examples
How to remove duplicate sentences across many articles in different files?
I have over 2k text files in 10 sub-folders. I would like to remove duplicate sentences from all text files.
Essentially, I need a tool that will open all files and search for sentences that match. Each similar sentences that are found is then deleted.
Input Sample:
this is sentence1, this is sentence1, this is sentence2

this is sentence1, this is sentence3
this is sentence4
Output Sample:
this is sentence1, 
this is sentence2

this is sentence3
this is sentence4
Hint: You need to Download and install "Replace Pioneer" on windows platform to finish following steps.
Assume you need to delete duplicate sentences under d:\test folder and sub-folder
1. ctrl-h open 'replace' dialog
* set 'search for pattern' to:

* set 'If' to:

click 'advanced' tab:
* set 'run following at the beginning of replace' to:

3. click "batch..." button to open "Batch Runner" window
4. click "search files" button
* set file name filter to 
* set search directory to parent folder 
* check option of 'search subdirectory' to allow searching sub-folder
* click "search" button, all files will be added to list, click "close".
5. click "batch replace" button, all files will have duplicate sentences removed.

* You can click the operation records in the middle of screen, to check what sentences are removed.
* You can set option of "change output folder" before step 5,to allow new files generate in new folder

Screenshot 1:  Replace_Window

Screenshot 2:  Replace_Advanced_Window

Similar Examples:
How to remove duplicate lines in multiple files in different sub-folders? (72%)
How to auto remove/delete consecutive duplicate sentences or phrases in a text file? (60%)
How to duplicate each line 5 times in a text file? (59%)
How to add in different sentences after the different lines? (58%)
How to save specified lines of text file into different files? (57%)
How to replace certain column from different files?  (56%)
How to split different groups of continuous numbers into different files? (56%)
How to remove partially duplicated lines in a text file? (56%)

Check Demo of Replace text in multiple files
duplicate sentences  to delete duplicate sentences  remove duplicate sentences  delete duplicate sentences  remove duplicate sentence  screen  duplicate sentence  subfolders  subfolder  records  check duplicate sentences  txt duplicate sentences search  batch remove duplicate records in a text file  remove duplicate sentences from text files  duplicate records  remove duplicate sentences from a text file  batch delete sentences  batch to delete text before