[lug] Bash shell script question

[Zan Lynx]

On 2/21/2018 7:40 PM, BC wrote:
> On 2/21/2018 7:20 PM, Zan Lynx wrote:
>> And there's potential garbage collection issues. It can surprise some
>> people that holding a byte slice of six characters can pin a 4K page.
>> This can happen a lot when parsing out of a file stream, then storing
>> that slice into a data structure without copying.
> 
> I grew up in the era when 4k of RAM was a lot.  But now...  A trifling 
> in the big scheme of things.

Well, it starts to add up. The situation I saw was a toy program I wrote 
that did something you can do with Awk, really. Read a big log file, 
maybe a few hundred megabytes? And store counts of how often words in 
field 6 appeared. And at the end output the list of words and counts.

It used a LOT more memory than I expected and when I profiled it I 
realized that when I thought I was being clever by storing byte slices, 
I had actually prevented the garbage collector from collecting any of 
the file buffers. I just changed it to store regular strings instead, 
which Go copies out of byte slices and the problem went away.