[lug] Speed up osm2pgsql

[Steve Sullivan]

Hi Lee,

Not sure if you want to go this direction, but on ebay
there are a lot of big servers for sale that could
put the whole problem in main memory, ending swapping.

For example, Dell R910 6 core with 256 GB RAM for about $800.

Steve

On Thu, Feb 08, 2018 at 12:08:35PM -0700, Lee Woodworth wrote:
> Populating a postgresql database for open street map uses 230GB
> of virtual address space. The swap device is very active with
> lots of writes at an average rate of 20MB/s.
> 
> The source parsing phase was going to take 2+ weeks using a
> regular disk for swap. Changing to an ssd speeds things but
> writes 1.4TB (actual device writes) out of a 75TB TBW. The
> wear level reported by smartctl went from 0 to 10 just for
> for this phase.
> 
> The post-processing phase is going very slow during a database
> sort where the db is on a raid1. Does anybody know of knobs to
> tweak to speed up larger-than-memory table sorts (100GB)? Or do we
> just need to use an ssd and bite the bullet on having to replace
> them frequently?
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety

-- 

========================================
Steve Sullivan      steve.sullivan at mathcom.com
720-587-7498        http://www.mathcom.com
========================================