Open source: Top issues in the top enterprise packages

1© 2016 Rogue Wave Software, Inc. All Rights Reserved. 1
Top open source lessons
for every enterprise
Episode 4:
Top issues in the top enterprise
packages

Bill Crowell
Enterprise architect, OpenLogic support
Rogue Wave Software
Vince Cox
Enterprise architect, OpenLogic support
Rogue Wave Software
Presenters

Poll #1
What percentage of your mission critical software is open source?
A: 0 to 25%
B: 26 to 50%
C: 51 to 75%
D: 75%

1. Introduction
2. Setting the context
3. Top issues: middleware + runtimes
4. Top issues: databases
5. Top issues: security
6. Conclusion
7. Q&A
Agenda

Who wrote LevelDB and what version
of ActiveMQ did it debut in?
What percentage of web server market
share does Apache HTTP Server hold?
Pop quiz

Introduction
What problems do our clients commonly run into?
From our experience…
• ActiveMQ High Availability Shared Databases Using NFS
• PostgreSQL Database Performance
• Request Header Vulnerabilities

Poll #2
What type of ActiveMQ persistence store do you use?
A: KahaDB
B: LevelDB
C: Replicated LevelDB
D: We don’t use message persistence with ActiveMQ
E: We don’t use ActiveMQ

Top issues:
middleware + runtimes

Scenario #1
Observing crashing or unresponsive broker
Typical configuration/requirements
• Master/slave network broker setup with NFS mount
• NFSv4 (SoftNAS) cloud-based shared file system available with
Amazon Web Services EC2 or GlusterFS
• Often a high-throughput requirement (10k messages/second of 1-3kb
message size)
One of the following problems…
• Master dies resulting in “no master” scenario
• Slave prematurely claims lock resulting in 2
master brokers leading to message loss
• Continuous master/slave re-election
• Increasing CPUs/memory doesn’t help
• Non-existent prior to production

Scenario #1 solution
Use SAN, Replicated LevelDB, or Pluggable Storage Lockers
Three solutions
• Use block-level iSCSI driver with Storage Area Network (SAN)
• Master-slave for HA and Replicated LevelDB managed by Zookeeper
• Pluggable Storage Lockers
More points
• Inherent flaws in OS-level filesystem locking mechanism
• Exclusive file locks work great with a SAN but is most expensive
• Replicated LevelDB requires a more configuration and a quorum of
nodes (replicas / 2 + 1)
• Pluggable Storage Locker/Lease Database Locker

Lease Database Locker
Points
• Master must renew lease before lease expires
• The lease period can be configured
• If not renewed, then the slave takes ownership of the lease
becoming the new master
More points
• Leased locks can survive database replica failovers
• Can be used with any JDBC-compliant database
• Make sure to uniquely name your brokers
• Keep master/slave clocks synchronized with NTP service
• Uniquely name your brokers. Use connection pooling

Poll #3
Do you use PostgreSQL?
A: We don’t use PostgreSQL
B: We don’t use PostgreSQL but plan on using it in the future
C: We use PostgreSQL as a mission-critical application database
D: We use PostgreSQL as part of another open source project
E: We are migrating away from PostgreSQL

Top issues:
databases

Scenario #2
Database performance is poor
Symptoms
• Sorting and querying take a long time
• One particular query or web page is hanging the database
• “Sorry, too many clients already” or connection pool is full
Where do I start?
• Localize if possible: Data center, network,
database or application server?
• When did it start
• What changes took place?

Identify and analyze
Quick checks
• top with ‘c’ command shows process ID, CPU, and memory
utilization
• ”iostat –x –m 5” reveals disk IO wait times
• ELK = Elasticsearch + Logstash + Kibana
• “EXPLAIN ANALYZE <SQL>” shows execution time and table
scans
Tools
• psql: SELECT pid, datname, usename, query FROM pg_stat_activity;
pid datname usename query
42102 jboss jboss SELECT pid, datname, usename,
query FROM pg_stat_activity;
42103 jboss jboss SELECT video FROM news where...;

Tools
• pg_stat_statments
www.postgresql.org/docs/current/static/pgstatstatements.html
Note: pg_stat_statements requires more shared memory
postgresql.conf:
shared_preload_libraries = ‘pg_stat_statements’
Server restart is required after enabling the shared library
Reset statistics: select pg_stat_reset();

Tools
More on PostgreSQL performance:
www.craigkerstiens.com/2013/01/10/more-on-postgres-
performance/
SELECT
(total_time / 1000 / 60) as total_minutes,
(total_time/calls) as average_time,
query
FROM pg_stat_statements
ORDER BY 1 DESC
LIMIT 100;
The Query
Total Query Time (in minutes)
Average Time (in milliseconds)

Tools
• pbBadger: dalibo.github.io/pgbadger/
• Requires Perl
pgBadger: dalibo.github.io/pgbadger/
postgresql.conf
• log_min_duration_statement = 0
• log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d ’
• log_checkpoints = on
• log_connections = on
• log_disconnections = on
• log_lock_waits = on
• log_temp_files = 0
After making changes run: SELECT pg_reload_conf();

Implement and test (…and repeat...)
Supporting points
• What if I don’t find the offending SQL or table?
• Start looking at the database configuration.
• Did you benchmark and tune?
More points
• Most parameters are automatically adjusted.
• We find that many people make minimal changes to the default
configuration.
• As always, remember to test any configuration changes in a non-prod
environment first, and implement changes incrementally.
• Do not make numerous configuration changes all at once.
• Systematically measure your performance tests. Use a tool like pgbench.

Configuration Settings in postgresql.conf
• max_connections: Maximum simultaneous connections to the
database (default is 100).
• shared_buffers: database cache size (default is 128MB). 25% of
total RAM. Windows-based should be 64-512MB.
• effective_cache_size: Tells the query planner how much RAM
there is to execute. 50-75% of total RAM.
• work_mem: Used for complex sorts (default is 4MB). Check if it is
uncommented and what the value is.
• maintenance_work_mem: Amount of memory for background
processes for pgdump, pgrestore, vacuum, indexing, and bulk loads
(default is 64MB). 256MB-1GB for large databases.
• checkpoint_segments: Maxiumum # of log file segments between
WAL checkpoints (default is 3).

Configuration Settings in postgresql.conf
• wal_buffers: Write ahead log buffer used for writing a transaction to
disk.
What should I really set these values to?
• pgTune: pgtune.leopard.in.ua/
SELECT name, current_setting(name), SOURCE FROM pg_settings WHERE
SOURCE NOT IN ('default', 'override');
PostgreSQL 9.0 High Performance
www.amazon.com/PostgreSQL-High-Performance-Gregory-
Smith/dp/184951030X/163-3733534-8577963

Poll #4
Are you using SSLv3?
A: Yes
B: No

Top issues:
security

Scenario #3
Pen testing reveals the application server is vulnerable
Supporting points
• Cache-control, Pragma, Expires headers are not set
• X-Powered-By reveals the application server type
More points
• A proxy can inject an intermediate
page and compromise the site
• Certain versions of JBoss allow little to
no control of request header
manipulation

Request header manipulation strategies
Supporting points
• Proxy requests through Apache HTTP Server before sending
them to the application server
• Mod_expire could be used to explicitly set them. Does not
guarantee header will be set
• Set org.apache.catalina.connector.X_POWERED_BY to false
More points
• Using Apache HTTP Server’s mod_header module is easiest and preferred
option
• mod_headers module allows manipulation before and after the request
• If this is not an option, then a filter can do the same function
• There should never be a situation where an application server is internet
facing without the protection of a web server in the DMZ

DMZ
• The front end should always be in the DMZ
• This should never be an application server
• Web servers are “far more” capable than application
servers in this role
Best practice
• Traditional 3-headed monster, Web/App/DB should always reside at
unique layers in the infrastructure
• There should be horizontal and/or vertical separation between them at
all times
• This provides greater opportunities for other security gear to potentially
sniff and sort out other possible vulnerabilities. Like HIDS/NIDS, etc.

HTTPD Mod_expires
• Mod_expires can set the max_age and expirations
• ExpiresByType text/html M604800 (Expires a week after
modification)
HTTPD Mod_Header
• Much more flexible than mod_expires
• Header echo ^KC (copies all request headers starting with KS to
response headers
• Header set TESTHEADER “Hi, The request ran in %D uSEC’s”
• Env variables can be set as well. Much more capable than
mod_expires

Header manipulation examples
HTTPD Mod_header Example
• <ifModule mod)headers.c>
• Header set DateTimeHEader “%D %t”
• </ifModule>
Header merge Cache-Control no-cache
Header merge Cache-Control no-store
Yields
Cache-Control: no-cache, no-store

HTTPD Mod_header Example
• Header set Set-Cookie testcookie “expr=-z %
{req:Cookie}”
• Header merge Cache-Control no-cache env=CGI
• Header merge Cache-Control no cache
end=NO_CACHE
• Head merge Cache-Control not-store env=NO_STORE
Non-exclusive conditions. If all are set: Cache-Control: no-
cache, no-store
Using append instead of merge would cause a duplicate
no-cache message

HTTPD Mod_header (Powerful module)
• Add, Append, echo, edit, merge, set setifempty, unset
• Allows setting of environment variables to use are
triggers
• Expressions can be set as well for more additional
control
Examples prove that web servers are well suited for this
type of work.

JBoss Wildfly : Header Manipulation
• You shouldn’t, but if you absolutely must
• You could modify system properties to override certain
values. Or set a filter.
• Newer versions of JBoss provide header modification
Example
• <system properties>
• <property
name=“org.apache.coyote.http11.Http11Protocol.SERVER
” value=“someserver”/>
• <system-properties>
• For each version of Jboss, you may need to use the CLI to determine if
the necessary keys are available in that version

• You can, but apache httpd is still a good accomplice
• Remove the following the alter the headers
• <filter-ref name=“x-powered-by-header”/>
• Performed in the undertow segment of standalone.xml
Example
• As newer versions of Wildfly emerge there is a bit more control of
headers
• This does not remove the need to have a web server in front
• A reverse proxy is is vital piece of application server security

• CLI command structure will provide a view of current header
manipulation capabilities Wildfly has
• More flexibility in each new release
• Wildfly’s capabilities are far behind that of apache’s
• Having this level of control at your reverse proxy just makes
the most sense
• These features have been requested for quite some time
from the user base.
• Not a replacement for a frontend DMZ based web server

Who wrote LevelDB and what version of
ActiveMQ did it debut in?
Written by Google:
Jeff Dean who also has contributed to MapReduce and Google Translate
Sanjay Ghemawat who also contributed to MapReduce and iCal which is a
popular calendar application in Unix/Linux.
LevelDB debuted in ActiveMQ 5.10.0 but was not production ready.
What percentage of web server market share
does Apache HTTP Server hold?
July 2016
Apache HTTP Server: 52%
nginx: 30.5%
Microsoft: 12%
Pop quiz answers

Conclusion

JBoss/Wildfly: Newer versions of Wildfly allow administrators to control
HTTP headers and older versions of JBoss can be augmented with a filter
or Apache HTTP Server's mod_header for equivalent functionality.
So much open source!
PostgreSQL: Use tools like pgBench and pgTune.
PostgreSQL: Troubleshooting tools like pg_stat_statements and
pgBadger can help narrow down database performance issues.
ActiveMQ: A Lease Database Locker is a viable alternative to
NFS where SAN is cost prohibitive. Consider Replicated
LevelDB with Apache Zookeeper for performance.

Our support

Q & A

Watch on demand
• Watch this webinar on demand
• Read the recap blog to see the results of the
polls and Q&A session

Follow up
For OpenLogic support customers:
OSS Radio
Get a free OSS support ticket to experience our expertise
roguewave.com/freeticket
Free open source newsletter:
roguewave.com/products/open-source-support/openupdate

Open source: Top issues in the top enterprise packages

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to Open source: Top issues in the top enterprise packages

Similar to Open source: Top issues in the top enterprise packages (20)

More from Rogue Wave Software

More from Rogue Wave Software (20)

Recently uploaded

Recently uploaded (20)

Open source: Top issues in the top enterprise packages

Editor's Notes