cutting a field to length with awk/sed [duplicate]
up vote
5
down vote
favorite
This question already has an answer here:
Unix Truncate Column in csv file
2 answers
I have a rather large bunch of files that contains several fields pipe-delimited.
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|1212121212121212121212121212|2|0|1000|70|33107||1|Event
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|2323232323232323232323232323|2|0|1000|70|33107||1|Event
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|3434343434343434343434343434|2|0|1000|70|33107||1|Event
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|4545454545454545454545454545|2|0|1000|70|33107||1|Event
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|5656565656565656565656565656|2|0|1000|70|33107||1|Event
Notice the eighth field. It currently has 29 characters and I'm supposed to trim it so it has only five characters left.
The only (convoluted) solution I've come up with is this:
Isolate the fields I want to trim:
awk -F "|" '{print $8}' > Original_Fields
Trim the fields
cp Original_Fields Temp
more Temp | cut -c -5 > Trimmed_Fields
Create a susbtitution script with sed
grep -rh -f <file_with_matching_strings> /path/to/files > Original_Strings
vi Original_Strings
:%s/^/grep -rl "/g
:%s/$/" /path/to/file | xargs sed -i 's//g
:wq!
And then edit the Original_Fields and Trimmed_Fields files, so I end up with
grep -rl /path/to/file | xargs sed -i 's/Original_Field/Trimmed_Field/g'
This works, but I strongly suspect there must be a quicker way to accomplish this with AWK and SED, so I can do all of this in just one step.
text-processing awk sed
edited Nov 27 at 9:04
ilkkachu
54.1k782147
asked Nov 27 at 8:13
Alan Mackey
282
New contributor
Alan Mackey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
marked as duplicate by Stéphane Chazelas awk
Users with the awk badge can single-handedly close awk questions as duplicates and reopen them as needed.
StackExchange.ready(function() {
if (StackExchange.options.isMobile) return;
$('.dupe-hammer-message-hover:not(.hover-bound)').each(function() {
var $hover = $(this).addClass('hover-bound'),
$msg = $hover.siblings('.dupe-hammer-message');
$hover.hover(
function() {
$hover.showInfoMessage('', {
messageElement: $msg.clone().show(),
transient: false,
position: { my: 'bottom left', at: 'top center', offsetTop: -7 },
dismissable: false,
relativeToBody: true
});
},
function() {
StackExchange.helpers.removeMessages();
}
);
});
});
Nov 27 at 13:07
This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.
add a comment |
up vote
5
down vote
favorite
This question already has an answer here:
Unix Truncate Column in csv file
2 answers
I have a rather large bunch of files that contains several fields pipe-delimited.
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|1212121212121212121212121212|2|0|1000|70|33107||1|Event
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|2323232323232323232323232323|2|0|1000|70|33107||1|Event
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|3434343434343434343434343434|2|0|1000|70|33107||1|Event
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|4545454545454545454545454545|2|0|1000|70|33107||1|Event
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|5656565656565656565656565656|2|0|1000|70|33107||1|Event
Notice the eighth field. It currently has 29 characters and I'm supposed to trim it so it has only five characters left.
The only (convoluted) solution I've come up with is this:
Isolate the fields I want to trim:
awk -F "|" '{print $8}' > Original_Fields
Trim the fields
cp Original_Fields Temp
more Temp | cut -c -5 > Trimmed_Fields
Create a susbtitution script with sed
grep -rh -f <file_with_matching_strings> /path/to/files > Original_Strings
vi Original_Strings
:%s/^/grep -rl "/g
:%s/$/" /path/to/file | xargs sed -i 's//g
:wq!
And then edit the Original_Fields and Trimmed_Fields files, so I end up with
grep -rl /path/to/file | xargs sed -i 's/Original_Field/Trimmed_Field/g'
This works, but I strongly suspect there must be a quicker way to accomplish this with AWK and SED, so I can do all of this in just one step.
text-processing awk sed
edited Nov 27 at 9:04
ilkkachu
54.1k782147
asked Nov 27 at 8:13
Alan Mackey
282
New contributor
Alan Mackey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
marked as duplicate by Stéphane Chazelas awk
Users with the awk badge can single-handedly close awk questions as duplicates and reopen them as needed.
StackExchange.ready(function() {
if (StackExchange.options.isMobile) return;
$('.dupe-hammer-message-hover:not(.hover-bound)').each(function() {
var $hover = $(this).addClass('hover-bound'),
$msg = $hover.siblings('.dupe-hammer-message');
$hover.hover(
function() {
$hover.showInfoMessage('', {
messageElement: $msg.clone().show(),
transient: false,
position: { my: 'bottom left', at: 'top center', offsetTop: -7 },
dismissable: false,
relativeToBody: true
});
},
function() {
StackExchange.helpers.removeMessages();
}
);
});
});
Nov 27 at 13:07
This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.
add a comment |
up vote
5
down vote
favorite
up vote
5
down vote
favorite
This question already has an answer here:
Unix Truncate Column in csv file
2 answers
I have a rather large bunch of files that contains several fields pipe-delimited.
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|1212121212121212121212121212|2|0|1000|70|33107||1|Event
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|2323232323232323232323232323|2|0|1000|70|33107||1|Event
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|3434343434343434343434343434|2|0|1000|70|33107||1|Event
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|4545454545454545454545454545|2|0|1000|70|33107||1|Event
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|5656565656565656565656565656|2|0|1000|70|33107||1|Event
Notice the eighth field. It currently has 29 characters and I'm supposed to trim it so it has only five characters left.
The only (convoluted) solution I've come up with is this:
Isolate the fields I want to trim:
awk -F "|" '{print $8}' > Original_Fields
Trim the fields
cp Original_Fields Temp
more Temp | cut -c -5 > Trimmed_Fields
Create a susbtitution script with sed
grep -rh -f <file_with_matching_strings> /path/to/files > Original_Strings
vi Original_Strings
:%s/^/grep -rl "/g
:%s/$/" /path/to/file | xargs sed -i 's//g
:wq!
And then edit the Original_Fields and Trimmed_Fields files, so I end up with
grep -rl /path/to/file | xargs sed -i 's/Original_Field/Trimmed_Field/g'
This works, but I strongly suspect there must be a quicker way to accomplish this with AWK and SED, so I can do all of this in just one step.
text-processing awk sed
edited Nov 27 at 9:04
ilkkachu
54.1k782147
asked Nov 27 at 8:13
Alan Mackey
282
New contributor
Alan Mackey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
This question already has an answer here:
Unix Truncate Column in csv file
2 answers
I have a rather large bunch of files that contains several fields pipe-delimited.
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|1212121212121212121212121212|2|0|1000|70|33107||1|Event
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|2323232323232323232323232323|2|0|1000|70|33107||1|Event
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|3434343434343434343434343434|2|0|1000|70|33107||1|Event
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|4545454545454545454545454545|2|0|1000|70|33107||1|Event
5595340959340|1|MXPYAQWE|870569689456954654|0|0|20181018224126|5656565656565656565656565656|2|0|1000|70|33107||1|Event
Notice the eighth field. It currently has 29 characters and I'm supposed to trim it so it has only five characters left.
The only (convoluted) solution I've come up with is this:
Isolate the fields I want to trim:
awk -F "|" '{print $8}' > Original_Fields
Trim the fields
cp Original_Fields Temp
more Temp | cut -c -5 > Trimmed_Fields
Create a susbtitution script with sed
grep -rh -f <file_with_matching_strings> /path/to/files > Original_Strings
vi Original_Strings
:%s/^/grep -rl "/g
:%s/$/" /path/to/file | xargs sed -i 's//g
:wq!
And then edit the Original_Fields and Trimmed_Fields files, so I end up with
grep -rl /path/to/file | xargs sed -i 's/Original_Field/Trimmed_Field/g'
This works, but I strongly suspect there must be a quicker way to accomplish this with AWK and SED, so I can do all of this in just one step.
This question already has an answer here:
Unix Truncate Column in csv file
2 answers
text-processing awk sed
text-processing awk sed
edited Nov 27 at 9:04
ilkkachu
54.1k782147
asked Nov 27 at 8:13
Alan Mackey
282
New contributor
Alan Mackey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited Nov 27 at 9:04
ilkkachu
54.1k782147
asked Nov 27 at 8:13
Alan Mackey
282
New contributor
Alan Mackey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited Nov 27 at 9:04
ilkkachu
54.1k782147
edited Nov 27 at 9:04
ilkkachu
54.1k782147
edited Nov 27 at 9:04
ilkkachu
54.1k782147
54.1k782147
asked Nov 27 at 8:13
Alan Mackey
282
New contributor
Alan Mackey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked Nov 27 at 8:13
Alan Mackey
282
asked Nov 27 at 8:13
Alan Mackey
282
282
New contributor
Alan Mackey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Alan Mackey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Alan Mackey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
marked as duplicate by Stéphane Chazelas awk
Users with the awk badge can single-handedly close awk questions as duplicates and reopen them as needed.
StackExchange.ready(function() {
if (StackExchange.options.isMobile) return;
$('.dupe-hammer-message-hover:not(.hover-bound)').each(function() {
var $hover = $(this).addClass('hover-bound'),
$msg = $hover.siblings('.dupe-hammer-message');
$hover.hover(
function() {
$hover.showInfoMessage('', {
messageElement: $msg.clone().show(),
transient: false,
position: { my: 'bottom left', at: 'top center', offsetTop: -7 },
dismissable: false,
relativeToBody: true
});
},
function() {
StackExchange.helpers.removeMessages();
}
);
});
});
Nov 27 at 13:07
This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.
marked as duplicate by Stéphane Chazelas awk
Users with the awk badge can single-handedly close awk questions as duplicates and reopen them as needed.
StackExchange.ready(function() {
if (StackExchange.options.isMobile) return;
$('.dupe-hammer-message-hover:not(.hover-bound)').each(function() {
var $hover = $(this).addClass('hover-bound'),
$msg = $hover.siblings('.dupe-hammer-message');
$hover.hover(
function() {
$hover.showInfoMessage('', {
messageElement: $msg.clone().show(),
transient: false,
position: { my: 'bottom left', at: 'top center', offsetTop: -7 },
dismissable: false,
relativeToBody: true
});
},
function() {
StackExchange.helpers.removeMessages();
}
);
});
});
Nov 27 at 13:07
This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
up vote
12
down vote
accepted
Yes, you can trim and rebuild each line with AWK:
awk -F'|' 'BEGIN { OFS = FS } { $8 = substr($8, 1, 5); print }'
This sets the input and output separators to “|”, and for each line of input, trims the eighth field to five characters at most, and prints all the fields (including the updated field).
answered Nov 27 at 8:16
Stephen Kitt
159k24353429
add a comment |
up vote
1
down vote
Since your file is fixed width — or at least it seems — you can do it counting columns to keep and discard in sed:
sed -r 's/(.{68}).{23}(.*)/12/' yourfile
answered Nov 27 at 9:21
neurino
1,22011322
add a comment |
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
12
down vote
accepted
Yes, you can trim and rebuild each line with AWK:
awk -F'|' 'BEGIN { OFS = FS } { $8 = substr($8, 1, 5); print }'
This sets the input and output separators to “|”, and for each line of input, trims the eighth field to five characters at most, and prints all the fields (including the updated field).
answered Nov 27 at 8:16
Stephen Kitt
159k24353429
add a comment |
up vote
12
down vote
accepted
Yes, you can trim and rebuild each line with AWK:
awk -F'|' 'BEGIN { OFS = FS } { $8 = substr($8, 1, 5); print }'
This sets the input and output separators to “|”, and for each line of input, trims the eighth field to five characters at most, and prints all the fields (including the updated field).
answered Nov 27 at 8:16
Stephen Kitt
159k24353429
add a comment |
up vote
12
down vote
accepted
up vote
12
down vote
accepted
Yes, you can trim and rebuild each line with AWK:
awk -F'|' 'BEGIN { OFS = FS } { $8 = substr($8, 1, 5); print }'
This sets the input and output separators to “|”, and for each line of input, trims the eighth field to five characters at most, and prints all the fields (including the updated field).
answered Nov 27 at 8:16
Stephen Kitt
159k24353429
Yes, you can trim and rebuild each line with AWK:
awk -F'|' 'BEGIN { OFS = FS } { $8 = substr($8, 1, 5); print }'
This sets the input and output separators to “|”, and for each line of input, trims the eighth field to five characters at most, and prints all the fields (including the updated field).
answered Nov 27 at 8:16
Stephen Kitt
159k24353429
answered Nov 27 at 8:16
Stephen Kitt
159k24353429
answered Nov 27 at 8:16
Stephen Kitt
159k24353429
answered Nov 27 at 8:16
Stephen Kitt
159k24353429
159k24353429
add a comment |
add a comment |
up vote
1
down vote
Since your file is fixed width — or at least it seems — you can do it counting columns to keep and discard in sed:
sed -r 's/(.{68}).{23}(.*)/12/' yourfile
answered Nov 27 at 9:21
neurino
1,22011322
add a comment |
up vote
1
down vote
Since your file is fixed width — or at least it seems — you can do it counting columns to keep and discard in sed:
sed -r 's/(.{68}).{23}(.*)/12/' yourfile
answered Nov 27 at 9:21
neurino
1,22011322
add a comment |
up vote
1
down vote
up vote
1
down vote
Since your file is fixed width — or at least it seems — you can do it counting columns to keep and discard in sed:
sed -r 's/(.{68}).{23}(.*)/12/' yourfile
answered Nov 27 at 9:21
neurino
1,22011322
Since your file is fixed width — or at least it seems — you can do it counting columns to keep and discard in sed:
sed -r 's/(.{68}).{23}(.*)/12/' yourfile
answered Nov 27 at 9:21
neurino
1,22011322
answered Nov 27 at 9:21
neurino
1,22011322
answered Nov 27 at 9:21
neurino
1,22011322
answered Nov 27 at 9:21
neurino
1,22011322
1,22011322
add a comment |
add a comment |
