[lug] Speed up osm2pgsql

Zan Lynx zlynx at acm.org
Thu Feb 8 16:53:57 MST 2018


On February 8, 2018 12:08:35 PM MST, Lee Woodworth <blug-mail at duboulder.com> wrote:
>Populating a postgresql database for open street map uses 230GB
>of virtual address space. The swap device is very active with
>lots of writes at an average rate of 20MB/s.
>
>The source parsing phase was going to take 2+ weeks using a
>regular disk for swap. Changing to an ssd speeds things but
>writes 1.4TB (actual device writes) out of a 75TB TBW. The
>wear level reported by smartctl went from 0 to 10 just for
>for this phase.
>
>The post-processing phase is going very slow during a database
>sort where the db is on a raid1. Does anybody know of knobs to
>tweak to speed up larger-than-memory table sorts (100GB)? Or do we
>just need to use an ssd and bite the bullet on having to replace
>them frequently?
>_______________________________________________
>Web Page:  http://lug.boulder.co.us
>Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
>Join us on IRC: irc.hackingsociety.org port=6667
>channel=#hackingsociety

I just thought of this but maybe an Optane P900 drive? For your swap. The drive endurance is much higher than Flash. It is lower latency too.

They are expensive but probably cheaper than 256 GB of RAM.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20180208/63225ac4/attachment.html>


More information about the LUG mailing list