[lug] Speed up osm2pgsql

Lee Woodworth blug-mail at duboulder.com
Fri Feb 9 02:09:43 MST 2018


There have been several interesting ideas. Unfortunately the loader
is 3rd party software (osm2pgsql from open street map) and changing
it is impractical right now. Major hardware expenses are something
we could consider next year.

Since changing the loader isn't possible now, I think I have to adjust
the execution environment.

The high level observations for the slow phase are that one thread out
of 4 is active using 5% of the cpu capacity. The main table space device
for the data tables and pg_tmp has a  constant stream of reads and writes,
~1200 tps according to iostat. The indexes use a different table space
that has minimal activity during this phase. I am thinking of trying these
to reduce I/O latency:

1) move the database to another system where postgres can be
    given 16GB of ram for shared buffers. This might help postgres
    schedule actual writes in a more sequential fashion
2) symlink pg_tmp to a directory on a different device than the
    data tables device to spread the i/o over two devices
3) acquire 500GB ssds for the index table space and pg_tmp


More information about the LUG mailing list